Quizz: Targeted crowdsourcing with a billion (potential) users

Abstract: We describe Quizz, a gamified crowdsourcing system that simultaneously assesses the knowledge of users and acquires new knowledge from them. Quizz operates by asking users to complete short quizzes on specific topics; as a user answers the quiz questions, Quizz estimates the user's competence. To acquire new knowledge, Quizz also incorporates questions for which we do not have a known answer; the answers given by competent users provide useful signals for selecting the correct answers for these questions. Quizz actively tries to identify knowledgeable users on the Internet by running advertising campaigns, effectively leveraging the targeting capabilities of existing, publicly available, ad placement services. Quizz quantifies the contributions of the users using information theory and sends feedback to the advertisingsystem about each user. The feedback allows the ad targeting mechanism to further optimize ad placement. Our experiments, which involve over ten thousand users, confirm that we can crowdsource knowledge curation for niche and specialized topics, as the advertising network can automatically identify users with the desired expertise and interest in the given topic. We present controlled experiments that examine the effect of various incentive mechanisms, highlighting the need for having short-term rewards as goals, which incentivize the users to contribute. Finally, our cost-quality analysis indicates that the cost of our approach is below that of hiring workers through paid-crowdsourcing platforms, while offering the additional advantage of giving access to billions of potential users all over the planet, and being able to reach users with specialized expertise that is not typically available through existing labor marketplaces.

Targeted Crowdsourcing through the Quizz Framework

The research presented in the paper explores Quizz, a novel gamified crowdsourcing system designed to assess and acquire new knowledge from a substantial and diverse user base. The system's primary objective is to identify and engage knowledgeable participants, crowdsource specialized knowledge curation, and offer an alternative to traditional paid-crowdsourcing models.

Key Concepts and Framework

The Quizz system operates by presenting users with topic-specific quizzes, which serve dual purposes: evaluating user competence and soliciting new knowledge via questions without known answers. Sophisticated algorithms estimate user expertise, while competent responses enhance the system’s collective knowledge base. A distinguishing feature of the Quizz system is its integration with existing internet advertising frameworks to actively identify and recruit knowledgeable users offset by targeted ad placements.

Key aspects of Quizz include:

• Knowledge Assessment: The system employs both calibration questions (with known answers to assess competence) and collection questions (to derive knowledge from user input).
• Recruitment of Experts: Quizz leverages modern ad placement services for effective user targeting, optimizing resource allocation by reinforcing user competence feedback loops into ad placement algorithms.

Empirical Evaluation and Results

The empirical assessment of the Quizz system involved controlled user trials, engaging over ten thousand individuals. The study's results convincingly demonstrate that the advertising system efficiently draws users with the requisite expertise, validating the core hypothesis that targeted crowdsourcing can be both more affordable and richly diverse than conventional paid models.

• Cost and Quality Analysis: Quizz's model demonstrated reduced costs compared to traditional paid-crowdsourcing while maintaining high-quality outputs from niche experts. The cost-quality analysis indicates significant efficiency gains without compromising the accuracy of information garnered.
• Incentive Mechanisms: Various incentive schemes were tested, revealing the necessity for immediate and short-term rewards to sustain user engagement while deterring low-quality participation.
• Optimization and Scalability: By employing a data-driven, constantly adjusting feedback system with ad targeting services, Quizz system enhances engagement indicators such as conversion rates from ad clicks and quality of contribution.

Implications and Future Directions

The implications of the Quizz system impact both theoretical and practical realms of AI and crowdsourcing. The framework illustrates how leveraging existing marketing infrastructures for targeting expertise can enhance the quality and economic viability of crowdsourcing operations. The integration of dynamic, data-informed adjustment mechanisms for user engagement and recruitment sets a precedent for evolving crowdsourcing strategies.

From a theoretical standpoint, the paper expands upon the exploration-exploitation paradigm and Bayesian frameworks, refining these concepts with empirical data. This can enhance the adaptive mechanics of ML models in future systems.

Further avenues for research may include enhancing the granularity of user expertise estimation and refining natural language processing techniques to remove reliance on pre-validated questions as standards of correctness. Integrating real-time feedback loops into AI-driven marketing platforms could further increase precision in targeting potentially high-value volunteers, advancing the scalability and robustness of systems like Quizz.

In conclusion, the Quizz system marks a significant contribution towards economizing the process of engaging and retaining unpaid contributors with specialized knowledge, offering a template for future endeavors in targeted crowdsourcing in a digital era.