Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A2C: A Modular Multi-stage Collaborative Decision Framework for Human-AI Teams (2401.14432v1)

Published 25 Jan 2024 in cs.HC and cs.LG

Abstract: This paper introduces A2C, a multi-stage collaborative decision framework designed to enable robust decision-making within human-AI teams. Drawing inspiration from concepts such as rejection learning and learning to defer, A2C incorporates AI systems trained to recognise uncertainty in their decisions and defer to human experts when needed. Moreover, A2C caters to scenarios where even human experts encounter limitations, such as in incident detection and response in cyber Security Operations Centres (SOC). In such scenarios, A2C facilitates collaborative explorations, enabling collective resolution of complex challenges. With support for three distinct decision-making modes in human-AI teams: Automated, Augmented, and Collaborative, A2C offers a flexible platform for developing effective strategies for human-AI collaboration. By harnessing the strengths of both humans and AI, it significantly improves the efficiency and effectiveness of complex decision-making in dynamic and evolving environments. To validate A2C's capabilities, we conducted extensive simulative experiments using benchmark datasets. The results clearly demonstrate that all three modes of decision-making can be effectively supported by A2C. Most notably, collaborative exploration by (simulated) human experts and AI achieves superior performance compared to AI in isolation, underscoring the framework's potential to enhance decision-making within human-AI teams.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. A research agenda for hybrid intelligence: augmenting human intellect with collaborative, adaptive, responsible, and explainable artificial intelligence. Computer 53, 08 (2020), 18–28.
  2. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 4183–4192.
  3. Leo Breiman. 2001. Random forests. Machine learning 45 (2001), 5–32.
  4. Human-AI Ensembles: When Can They Work? Journal of Management (2023), 01492063231194968.
  5. C Chow. 1970. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory 16, 1 (1970), 41–46.
  6. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning 20 (1995), 273–297.
  7. CriticalStart.com. 2019. The Impact of Security Alert Overload. White Paper. CriticalStart.com. www.criticalstart.com/resources/research-report-the-impact-of-security-alert-overload
  8. Chris Crowley and Barbara Filkins. 2022. SANS 2022 SOC Survey. White Paper. Escal Institute of Advanced Technologies (SANS Institute). www.sans.org/white-papers/sans-2022-soc-survey
  9. Regression under human assistance. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 2611–2620.
  10. Anna Fleck. 2022. Cybercrime Expected To Skyrocket in Coming Years – Statista’s Cybersecurity Outlook. https://www.statista.com/chart/28878/expected-cost-of-cybercrime-until-2027 Accessed: 9-April-2023.
  11. Considerations for Human-Machine Teaming in Cybersecurity. In Augmented Cognition: 13th International Conference, AC 2019, Held as Part of the 21st HCI International Conference, HCII 2019, Orlando, FL, USA, July 26–31, 2019, Proceedings 21. Springer, 153–168.
  12. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1705–1714.
  13. Generative adversarial networks. Commun. ACM 63, 11 (2020), 139–144.
  14. DROCC: Deep robust one-class classification. In International conference on machine learning. PMLR, 3711–3721.
  15. Machine Learning with a Reject Option: A survey. ArXiv abs/2107.11277 (2021).
  16. Multilayer feedforward networks are universal approximators. Neural Networks 2, 5 (1989), 359–366. https://doi.org/10.1016/0893-6080(89)90020-8
  17. Towards unbiased and accurate deferral to multiple experts. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 154–165.
  18. Transformers in vision: A survey. ACM computing surveys (CSUR) 54, 10s (2022), 1–41.
  19. Learning multiple layers of features from tiny images. (2009).
  20. ImageNet Classification with Deep Convolutional Neural Networks. Commun. ACM 60, 6 (may 2017), 84–90. https://doi.org/10.1145/3065386
  21. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324.
  22. Explainable deep one-class classification. arXiv preprint arXiv:2007.01760 (2020).
  23. Opportunities and challenges for human-machine teaming in cybersecurity operations. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 63. SAGE Publications Sage CA: Los Angeles, CA, 442–446.
  24. Predict responsibly: improving fairness and accuracy by learning to defer. Advances in Neural Information Processing Systems 31 (2018).
  25. Samaneh Mahdavifar and Ali A Ghorbani. 2019. Application of deep learning to cybersecurity: A survey. Neurocomputing 347 (2019), 149–176.
  26. MA Majid and K Ariffi. 2019. Success factors for cyber security operation center (SOC) establishment. In Proc. 1st Int. Conf. Informat., Eng., Sci. Technol. 1–11.
  27. Hussein Mozannar and David Sontag. 2020. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning. PMLR, 7076–7087.
  28. Human-AI Teaming: State-of-the-Art and Research Needs. (2021).
  29. Cecile Paris and Andrew Reeson. 2021. What’s the secret to making sure AI doesn’t steal your job? Work with it, not against it.
  30. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442 (2023).
  31. A survey on deep learning: Algorithms, techniques, and applications. ACM Computing Surveys (CSUR) 51, 5 (2018), 1–36.
  32. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv:1903.12220 (2019).
  33. Panda: Adapting pretrained features for anomaly detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2806–2814.
  34. Deep one-class classification. In International conference on machine learning. PMLR, 4393–4402.
  35. Multiresolution knowledge distillation for anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 14902–14912.
  36. Estimating the support of a high-dimensional distribution. Neural computation 13, 7 (2001), 1443–1471.
  37. On Subset Selection of Multiple Humans To Improve Human-AI Team Accuracy. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems. 317–325.
  38. Multi-Class Anomaly Detection. In International Conference on Neural Information Processing. Springer, 359–371.
  39. KDD Cup 1999 Data. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C51C7N.
  40. Security operations center: A systematic study and open challenges. IEEE Access 8 (2020), 227756–227779.
  41. Deep learning for identifying metastatic breast cancer. arXiv preprint arXiv:1606.05718 (2016).
  42. David D Woods. 2016. The risks of autonomy: Doyle’s catch. Journal of Cognitive Engineering and Decision Making 10, 2 (2016), 131–133.
  43. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017).
  44. A Unified Model for Multi-class Anomaly Detection. arXiv:2206.03687 [cs.CV]
  45. Old is gold: Redefining the adversarially learned one-class classifier training paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14183–14193.
  46. A survey on learning to reject. Proc. IEEE 111, 2 (2023), 185–215.
  47. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International conference on learning representations.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Shahroz Tariq (20 papers)
  2. Mohan Baruwal Chhetri (7 papers)
  3. Surya Nepal (115 papers)
  4. Cecile Paris (34 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets