Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Challenge AI Mind: A Crowd System for Proactive AI Testing (1810.09030v1)

Published 21 Oct 2018 in cs.AI

Abstract: AI has burrowed into our lives in various aspects; however, without appropriate testing, deployed AI systems are often being criticized to fail in critical and embarrassing cases. Existing testing approaches mainly depend on fixed and pre-defined datasets, providing a limited testing coverage. In this paper, we propose the concept of proactive testing to dynamically generate testing data and evaluate the performance of AI systems. We further introduce Challenge.AI, a new crowd system that features the integration of crowdsourcing and machine learning techniques in the process of error generation, error validation, error categorization, and error analysis. We present experiences and insights into a participatory design with AI developers. The evaluation shows that the crowd workflow is more effective with the help of machine learning techniques. AI developers found that our system can help them discover unknown errors made by the AI models, and engage in the process of proactive testing.

User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Siwei Fu (11 papers)
  2. Anbang Xu (10 papers)
  3. Xiaotong Liu (16 papers)
  4. Huimin Zhou (2 papers)
  5. Rama Akkiraju (9 papers)
Citations (1)

Summary

We haven't generated a summary for this paper yet.