2000 character limit reached
Analysing and Organising Human Communications for AI Fairness-Related Decisions: Use Cases from the Public Sector (2404.00022v1)
Published 20 Mar 2024 in cs.HC and cs.AI
Abstract: AI algorithms used in the public sector, e.g., for allocating social benefits or predicting fraud, often involve multiple public and private stakeholders at various phases of the algorithm's life-cycle. Communication issues between these diverse stakeholders can lead to misinterpretation and misuse of algorithms. We investigate the communication processes for AI fairness-related decisions by conducting interviews with practitioners working on algorithmic systems in the public sector. By applying qualitative coding analysis, we identify key elements of communication processes that underlie fairness-related human decisions. We analyze the division of roles, tasks, skills, and challenges perceived by stakeholders. We formalize the underlying communication issues within a conceptual framework that i. represents the communication patterns ii. outlines missing elements, such as actors who miss skills for their tasks. The framework is used for describing and analyzing key organizational issues for fairness-related decisions. Three general patterns emerge from the analysis: 1. Policy-makers, civil servants, and domain experts are less involved compared to developers throughout a system's life-cycle. This leads to developers taking on extra roles such as advisor, while they potentially miss the required skills and guidance from domain experts. 2. End-users and policy-makers often lack the technical skills to interpret a system's limitations, and rely on developer roles for making decisions concerning fairness issues. 3. Citizens are structurally absent throughout a system's life-cycle, which may lead to decisions that do not include relevant considerations from impacted stakeholders.
- Rodolfa, K.T., Salomon, E., Haynes, L., Mendieta, I.H., Larson, J., Ghani, R.: Case study: Predictive fairness to reduce misdemeanor recidivism through social service interventions. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 142–153. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372863 . https://doi.org/10.1145/3351095.3372863 Williamson [2016] Williamson, B.: Digital education governance: data visualization, predictive analytics, and ‘real-time’policy instruments. Journal of education policy 31(2), 123–141 (2016) Van Veenstra et al. [2019] Van Veenstra, A.F.E., Djafari, S., Grommé, F., Kotterink, B., Baartmans, R.F.W.: Quickscan AI in the Publieke dienstverlening (2019). http://resolver.tudelft.nl/uuid:be7417ac-7829-454c-9eb8-687d89c92dce Hoekstra et al. [2021] Hoekstra, Chideock, Veenstra, V.: TNO Rapportage Quickscan AI in the Publieke sector II (2021). https://www.rijksoverheid.nl/documenten/rapporten/2021/05/20/quickscan-ai-in-publieke-dienstverlening-ii Mehrabi et al. [2021] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021) Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Williamson, B.: Digital education governance: data visualization, predictive analytics, and ‘real-time’policy instruments. Journal of education policy 31(2), 123–141 (2016) Van Veenstra et al. [2019] Van Veenstra, A.F.E., Djafari, S., Grommé, F., Kotterink, B., Baartmans, R.F.W.: Quickscan AI in the Publieke dienstverlening (2019). http://resolver.tudelft.nl/uuid:be7417ac-7829-454c-9eb8-687d89c92dce Hoekstra et al. [2021] Hoekstra, Chideock, Veenstra, V.: TNO Rapportage Quickscan AI in the Publieke sector II (2021). https://www.rijksoverheid.nl/documenten/rapporten/2021/05/20/quickscan-ai-in-publieke-dienstverlening-ii Mehrabi et al. [2021] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021) Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Van Veenstra, A.F.E., Djafari, S., Grommé, F., Kotterink, B., Baartmans, R.F.W.: Quickscan AI in the Publieke dienstverlening (2019). http://resolver.tudelft.nl/uuid:be7417ac-7829-454c-9eb8-687d89c92dce Hoekstra et al. [2021] Hoekstra, Chideock, Veenstra, V.: TNO Rapportage Quickscan AI in the Publieke sector II (2021). https://www.rijksoverheid.nl/documenten/rapporten/2021/05/20/quickscan-ai-in-publieke-dienstverlening-ii Mehrabi et al. [2021] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021) Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hoekstra, Chideock, Veenstra, V.: TNO Rapportage Quickscan AI in the Publieke sector II (2021). https://www.rijksoverheid.nl/documenten/rapporten/2021/05/20/quickscan-ai-in-publieke-dienstverlening-ii Mehrabi et al. [2021] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021) Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021) Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Williamson, B.: Digital education governance: data visualization, predictive analytics, and ‘real-time’policy instruments. Journal of education policy 31(2), 123–141 (2016) Van Veenstra et al. [2019] Van Veenstra, A.F.E., Djafari, S., Grommé, F., Kotterink, B., Baartmans, R.F.W.: Quickscan AI in the Publieke dienstverlening (2019). http://resolver.tudelft.nl/uuid:be7417ac-7829-454c-9eb8-687d89c92dce Hoekstra et al. [2021] Hoekstra, Chideock, Veenstra, V.: TNO Rapportage Quickscan AI in the Publieke sector II (2021). https://www.rijksoverheid.nl/documenten/rapporten/2021/05/20/quickscan-ai-in-publieke-dienstverlening-ii Mehrabi et al. [2021] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021) Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Van Veenstra, A.F.E., Djafari, S., Grommé, F., Kotterink, B., Baartmans, R.F.W.: Quickscan AI in the Publieke dienstverlening (2019). http://resolver.tudelft.nl/uuid:be7417ac-7829-454c-9eb8-687d89c92dce Hoekstra et al. [2021] Hoekstra, Chideock, Veenstra, V.: TNO Rapportage Quickscan AI in the Publieke sector II (2021). https://www.rijksoverheid.nl/documenten/rapporten/2021/05/20/quickscan-ai-in-publieke-dienstverlening-ii Mehrabi et al. [2021] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021) Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hoekstra, Chideock, Veenstra, V.: TNO Rapportage Quickscan AI in the Publieke sector II (2021). https://www.rijksoverheid.nl/documenten/rapporten/2021/05/20/quickscan-ai-in-publieke-dienstverlening-ii Mehrabi et al. [2021] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021) Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021) Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Van Veenstra, A.F.E., Djafari, S., Grommé, F., Kotterink, B., Baartmans, R.F.W.: Quickscan AI in the Publieke dienstverlening (2019). http://resolver.tudelft.nl/uuid:be7417ac-7829-454c-9eb8-687d89c92dce Hoekstra et al. [2021] Hoekstra, Chideock, Veenstra, V.: TNO Rapportage Quickscan AI in the Publieke sector II (2021). https://www.rijksoverheid.nl/documenten/rapporten/2021/05/20/quickscan-ai-in-publieke-dienstverlening-ii Mehrabi et al. [2021] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021) Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hoekstra, Chideock, Veenstra, V.: TNO Rapportage Quickscan AI in the Publieke sector II (2021). https://www.rijksoverheid.nl/documenten/rapporten/2021/05/20/quickscan-ai-in-publieke-dienstverlening-ii Mehrabi et al. [2021] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021) Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021) Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Hoekstra, Chideock, Veenstra, V.: TNO Rapportage Quickscan AI in the Publieke sector II (2021). https://www.rijksoverheid.nl/documenten/rapporten/2021/05/20/quickscan-ai-in-publieke-dienstverlening-ii Mehrabi et al. [2021] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021) Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021) Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021) Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008) Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360 European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720 Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305 Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019) Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020) Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577 Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022) Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022) Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022) Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022) Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013) Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999) Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999) Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992) Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Latour, B.: On technical mediation. Common knowledge 3(2) (1994) Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740 Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022) Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021) Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011) Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168 Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2 Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016) Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022) Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022) Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833 Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023) Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018) Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019) Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987) Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009 Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022) Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019) Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021) Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)
- Mirthe Dankloff (1 paper)
- Vanja Skoric (1 paper)
- Giovanni Sileno (11 papers)
- Sennay Ghebreab (7 papers)
- Emma Beauxis-Aussalet (4 papers)
- Jacco van Ossenbruggen (12 papers)