Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Complementarity in Human-AI Collaboration: Concept, Sources, and Evidence (2404.00029v2)

Published 21 Mar 2024 in cs.HC and cs.AI

Abstract: AI has the potential to significantly enhance human performance across various domains. Ideally, collaboration between humans and AI should result in complementary team performance (CTP) -- a level of performance that neither of them can attain individually. So far, however, CTP has rarely been observed, suggesting an insufficient understanding of the principle and the application of complementarity. Therefore, we develop a general concept of complementarity and formalize its theoretical potential as well as the actual realized effect in decision-making situations. Moreover, we identify information and capability asymmetry as the two key sources of complementarity. Finally, we illustrate the impact of each source on complementarity potential and effect in two empirical studies. Our work provides researchers with a comprehensive theoretical foundation of human-AI complementarity in decision-making and demonstrates that leveraging these sources constitutes a viable pathway towards designing effective human-AI collaboration, i.e., the realization of CTP.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (104)
  1. Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access, 6:52138–52160.
  2. Does explainable artificial intelligence improve human decision-making? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6618–6626.
  3. Beyond accuracy: The role of mental models in human-AI team performance. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 7(1):2–11.
  4. Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–16.
  5. Expl(AI)n it to me–explainable AI and information systems research. Business & Information Systems Engineering, 63(2):79–82.
  6. Expl(AI)ned: The impact of explainable artificial intelligence on users’ information processing. Information Systems Research, 34(4):1582–1602.
  7. Please take over: XAI, delegation of authority, and domain knowledge. Preprint, pages 1–41.
  8. Database models and managerial intuition: 50% model+ 50% manager. Management Science, 36(8):887–899.
  9. Breiman, L. (2001). Random forests. Machine Learning, 45(1):5–32.
  10. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, pages 1–155.
  11. Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems. In Proceedings of the International Conference on Intelligent User Interfaces, pages 454–464.
  12. Feature-based explanations don’t help people detect misclassifications of online toxicity. In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pages 95–106.
  13. To be or not to be …human? Theorizing the role of human-like competencies in conversational artificial intelligence agents. Journal of Management Information Systems, 39(4):969–1005.
  14. What makes teams work: Group effectiveness research from the shop floor to the executive suite. Journal of Management, 23(3):239–290.
  15. AI robo-advisor with big data analytics for financial services. In International Conference on Advances in Social Networks Analysis and Mining, pages 1027–1031.
  16. Hybrid intelligence. Business & Information Systems Engineering, 61(5):637–643.
  17. Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1):114–126.
  18. Human-algorithm collaboration: Achieving complementarity and avoiding unfairness. In Proceedings of the Conference on Fairness, Accountability, and Transparency, page 1639–1656.
  19. Dougherty, D. (1992). Interpretive barriers to successful product innovation in large firms. Organization Science, 3(2):179–202.
  20. Endsley, M. R. (2023). Supporting human-AI teams: Transparency, explainability, and situation awareness. Computers in Human Behavior, 140:1–16.
  21. G* power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2):175–191.
  22. Computation noise in human learning and decision-making: Origin, impact, function. Current Opinion in Behavioral Sciences, 38:124–132.
  23. Will humans-in-the-loop become borgs? Merits and pitfalls of working with AI. Management Information Systems Quarterly, 45(3):1527–1556.
  24. Cognitive challenges in human–artificial intelligence collaboration: Investigating the path toward productive delegation. Information Systems Research, 33(2):678–696.
  25. Exploring User Heterogeneity in Human Delegation Behavior towards AI. In Proceedings of the International Conference on Information Systems, pages 1–9.
  26. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673.
  27. Partial success in closing the gap between human and machine vision. In Advances in Neural Information Processing Systems, volume 34, pages 23885–23899.
  28. Conceptualization of the human-machine symbiosis–a literature review. In Proceedings of the Hawaii International Conference on System Sciences, pages 289–298.
  29. A new era: Artificial intelligence and machine learning in prostate cancer. Nature Reviews Urology, 16(7):391–403.
  30. Judgmental forecasts of time series affected by special events: Does providing a statistical forecast improve accuracy? Journal of Behavioral Decision Making, 12(1):37–53.
  31. Reconstructing constructivism: Causal models, bayesian learning mechanisms, and the theory theory. Psychological Bulletin, 138(6):1085–1108.
  32. The principles and limits of algorithm-in-the-loop decision making. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1–24.
  33. Augmenting the algorithm: Emerging human-in-the-loop work configurations. The Journal of Strategic Information Systems, 29(2):1–16.
  34. Grootswagers, T. (2020). A primer on running human behavioural experiments online. Behavior Research Methods, 52(6):2283–2286.
  35. Knowing about knowing: An illusion of human competence can hinder appropriate reliance on AI systems. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–18.
  36. Forming effective human-AI teams: Building machine learning models that complement the capabilities of multiple experts. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 2478–2484.
  37. Human-AI complementarity in hybrid intelligence systems: A structured literature review. In Proceedings of the Pacific Asia Conference on Information Systems, pages 1–14.
  38. Hillman, N. L. (2019). The use of artificial intelligence in gauging the risk of recidivism. Judges’ Journal, 58(1):36–39.
  39. Horwitz, S. K. (2005). The compositional impact of team diversity on performance: Theoretical considerations. Human Resource Development Review, 4(2):219–245.
  40. Densely connected convolutional networks. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 4700–4708.
  41. Huysman, M. (2020). Information systems research on artificial intelligence and work: A commentary on “Robo-Apocalypse cancelled? Reframing the automation and future of work debate”. Journal of Information Technology, 35(4):307–309.
  42. Eliciting human judgment for prediction algorithms. Management Science, 67(4):2314–2325.
  43. Editorial for the special section on humans, algorithms, and augmented intelligence: The future of work, organizations, and society. Information Systems Research, 32(3):675–687.
  44. Jarrahi, M. H. (2018). Artificial intelligence and the future of work: Human-AI symbiosis in organizational decision making. Business Horizons, 61(4):577–586.
  45. Why are we averse towards algorithms? A comprehensive literature review on algorithm aversion. In Proceedings of the European Conference on Information Systems, pages 1–16.
  46. Augmenting medical diagnosis decisions? an investigation into physicians’ decision-making process with artificial intelligence. Information Systems Research, 32(3):713–735.
  47. Kaggle (2019). House prices and images - SoCal. https://www.kaggle.com/ted8080/house-prices-and-images-socal (accessed: 2021-08-01).
  48. Evaluating human behaviour in response to AI recommendations for judgemental forecasting. European Journal of Operational Research, 303(3):1151–1167.
  49. Human decisions and machine predictions. The Quarterly Journal of Economics, 133(1):237–293.
  50. Algorithmic bias: Review, synthesis, and future research directions. European Journal of Information Systems, 31(3):388–409.
  51. Artificial intelligence and machine learning. Electronic Markets, 32(4):2235–2244.
  52. Let me explain: Impact of personal and impersonal explanations on trust in recommender systems. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–12.
  53. Hidden benefits of reward: A field experiment on motivation and monetary incentives. European Economic Review, 76:188–199.
  54. Towards a science of human-AI decision making: An overview of design space in empirical human-subject studies. In Proceedings of the Conference on Fairness, Accountability, and Transparency, page 1369–1385.
  55. "Why is ’Chicago’ deceptive?" Towards building model-driven tutorials for humans. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–13.
  56. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:1–72.
  57. Judgmental forecasting: A review of progress over the last 25 years. International Journal of Forecasting, 22(3):493–518.
  58. Deep learning. Nature, 521(7553):436–444.
  59. Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology, 49(4):764–766.
  60. Why we needn’t fear the machines: Opportunities for medicine in a machine learning world. Academic Medicine, 94(5):623–625.
  61. Licklider, J. C. R. (1960). Man-computer symbiosis. IRE Transactions on Human Factors in Electronics, HFE-1(1):4–11.
  62. Judgemental adjustment of initial forecasts: Its effectiveness and biases. Journal of Behavioral Decision Making, 8(3):149–168.
  63. Understanding the effect of out-of-distribution examples and interactive explanations on human-AI decision making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–45.
  64. Resistance to medical artificial intelligence. Journal of Consumer Research, 46(4):629–650.
  65. Predict responsibly: Improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems, volume 31, pages 1–11.
  66. What influences algorithmic decision-making? A systematic literature review on algorithm aversion. Technological Forecasting and Social Change, 175:1–26.
  67. Do I look like a criminal? Examining how race presentation impacts human judgement of recidivism. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–13.
  68. A test for evaluating performance in human-AI systems. Preprint (Version 1) available at Research Square, pages 1–12.
  69. International evaluation of an AI system for breast cancer screening. Nature, 577(7788):89–94.
  70. Meehl, P. E. (1957). When shall we use our heads instead of the formula? Journal of Counseling Psychology, 4(4):268–273.
  71. Thinking responsibly about responsible AI and ‘the dark side’ of AI. European Journal of Information Systems, 31(3):257–268.
  72. Artificial intelligence capability: Conceptualization, measurement calibration, and empirical study on its impact on organizational creativity and firm performance. Information & Management, 58(3):1–20.
  73. Consistent estimators for learning to defer to an expert. In Proceedings of the International Conference on Machine Learning, volume 119, pages 7076–7087.
  74. A taxonomy of human and ML strengths in decision-making to investigate human-ML complementarity. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 11(1):127–139.
  75. Experimental evidence of effective human–AI collaboration in medical decision-making. Scientific Reports, 12(1):1–10.
  76. "Why should I trust you?" Explaining the predictions of any classifier. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144.
  77. Anchors: High-precision model-agnostic explanations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, pages 1527–1535.
  78. Algorithmic decision-making and system destructiveness: A case of automatic debt recovery. European Journal of Information Systems, 31(3):313–338.
  79. The programmer’s assistant: Conversational interaction with a large language model for software development. In Proceedings of the International Conference on Intelligent User Interfaces, pages 491–514.
  80. Alternatives to the median absolute deviation. Journal of the American Statistical Association, 88(424):1273–1283.
  81. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252.
  82. On knowing when to switch from quantitative to judgemental forecasts. International Journal of Operations & Production Management, 11(6):27–37.
  83. Bringing judgment into combination forecasts. Journal of Operations Management, 13(4):311–321.
  84. Judgmental adjustment of statistical forecasts. In Principles of Forecasting: A Handbook for Researchers and Practitioners, pages 405–416.
  85. A robust inference method for decision-making in networks. Management Information Systems Quarterly, 46(2):713–738.
  86. Schunk, D. H. (1995). Self-efficacy, motivation, and performance. Journal of Applied Sport Psychology, 7(2):112–137.
  87. Machines as teammates: A research agenda on AI in team collaboration. Information & Management, 57(2):1–22.
  88. The recalcitrance of overconfidence and its contribution to decision aid neglect. Journal of Behavioral Decision Making, 18(1):29–53.
  89. Making use of difference: Diversity, debate, and decision comprehensiveness in top management teams. Academy of Management Journal, 42(6):662–673.
  90. AI for in-line vehicle sequence controlling: Development and evaluation of an adaptive machine learning artifact to predict sequence deviations in a mixed-model production line. Flexible Services and Manufacturing Journal, 34(3):709–747.
  91. Bayesian modeling of human–AI complementarity. Proceedings of the National Academy of Sciences, 119(11):1–7.
  92. How to grow a mind: Statistics, structure, and abstraction. Science, 331(6022):1279–1285.
  93. Terveen, L. G. (1995). Overview of human-computer collaboration. Knowledge-Based Systems, 8(2):67–81.
  94. Decision Support and Business Intelligence Systems. Pearson.
  95. Peer-to-peer loan acceptance and default prediction with artificial intelligence. Royal Society Open Science, 7(6):1–17.
  96. Evaluating XAI: A comparison of rule-based and example-based explanations. Artificial Intelligence, 291:1–19.
  97. Developing human/AI interactions for chat-based customer services: Lessons learned from the Norwegian government. European Journal of Information Systems, 32(1):10–22.
  98. Designing transparency for effective human-AI collaboration. Information Systems Frontiers, 24(3):877–895.
  99. Learning to complement humans. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 1526–1533.
  100. Do I trust my machine teammate? An investigation from perception to decision. In Proceedings of the International Conference on Intelligent User Interfaces, pages 460–468.
  101. You complete me: Human-AI teams and complementary expertise. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–28.
  102. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 295–305.
  103. Hybrid-augmented intelligence: Collaboration and cognition. Frontiers of Information Technology & Electronic Engineering, 18(2):153–179.
  104. Intelligence augmentation: Towards building human-machine symbiotic relationship. AIS Transactions on Human-Computer Interaction, 13(2):243–264.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Patrick Hemmer (19 papers)
  2. Max Schemmer (27 papers)
  3. Niklas Kühl (94 papers)
  4. Michael Vössing (23 papers)
  5. Gerhard Satzger (29 papers)
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets