Gamified crowd-sourcing of high-quality data for visual fine-tuning

Abstract: This paper introduces Gamified Adversarial Prompting (GAP), a framework that crowd-sources high-quality data for visual instruction tuning of large multimodal models. GAP transforms the data collection process into an engaging game, incentivizing players to provide fine-grained, challenging questions and answers that target gaps in the model's knowledge. Our contributions include (1) an approach to capture question-answer pairs from humans that directly address weaknesses in a model's knowledge, (2) a method for evaluating and rewarding players that successfully incentivizes them to provide high-quality submissions, and (3) a scalable, gamified platform that succeeds in collecting this data from over 50,000 participants in just a few weeks. Our implementation of GAP has significantly improved the accuracy of a small multimodal model, namely MiniCPM-Llama3-V-2.5-8B, increasing its GPT score from 0.147 to 0.477 on our dataset, approaching the benchmark set by the much larger GPT-4V. Moreover, we demonstrate that the data generated using MiniCPM-Llama3-V-2.5-8B also enhances its performance across other benchmarks, and exhibits cross-model benefits. Specifically, the same data improves the performance of QWEN2-VL-2B and QWEN2-VL-7B on the same multiple benchmarks.

• The paper introduces GAP, a framework that crowdsources data via gamification to expose model weaknesses and enhance visual instruction tuning.
• It demonstrates significant performance gains, with the GPT score rising from 0.147 to 0.477 and improvements observed across multiple models.
• The scalable platform engaged over 50,000 participants, highlighting its effectiveness and potential for advancing LMM capabilities.

Overview of "Gamified crowd-sourcing of high-quality data for visual fine-tuning"

This paper introduces "Gamified Adversarial Prompting (GAP)," a framework designed to crowd-source high-quality data aimed at enhancing visual instruction tuning in large multimodal models (LMMs). The method integrates gamification to motivate participants to create challenging question-answer pairs that reveal weaknesses in a model's understanding. The core contributions include: capturing valuable human-generated data, implementing an evaluation and reward system that encourages high-quality input, and developing a scalable platform that engaged over 50,000 participants in a brief period.

Key Concepts and Model Details

The paper focuses on Visual Question Answering (VQA), a crucial domain in AI, which enhances models' capabilities in understanding and reasoning about visual data. Despite LMMs' advancements in VQA, challenges persist in handling fine-grained details and complex reasoning. Visual instruction tuning, although beneficial, is constrained by the quality of its training data.

GAP Framework: The GAP framework transforms the data collection process into an engaging game. Players interact with images, posing questions that they think the model might answer incorrectly. By focusing on model weaknesses, GAP facilitates the creation of informative, adversarial examples crucial for model improvement.

Implementation and Results: GAP achieves significant results by fine-tuning the MiniCPM-Llama3-V-2.5-8B model with the generated dataset, elevating its GPT score from 0.147 to 0.477 on the newly created dataset. This marks a substantial performance enhancement, narrowing the gap with larger models such as GPT-4V. Furthermore, the acquired data improves other models, like QWEN2-VL-2B and QWEN2-VL-7B, across multiple benchmarks, underpinning its cross-model benefits.

Methodology

The paper meticulously details its strategy:

1. Datasets: Utilizes two distinct datasets derived from MS-COCO. The tainted set contains simpler images, designed for models to predict incorrectly upon instruction. The untainted set offers more complex images, challenging the model without manipulation.
2. Evaluation and Reward System: A sophisticated evaluation system inspired by reCAPTCHA analyzes player inputs, rewarding effective identification of model mistakes while discouraging incorrect markings. The reward system leverages both intrinsic and extrinsic motivations, including points, leaderboards, cash prizes, and future cryptocurrency rewards.
3. Experimental Design: The GAP-VQA dataset is curated for enhancing model performance. The experiments assess the dataset's impact on the base model and investigate its transferability across datasets and models, with positive outcomes.

Implications and Future Directions

The framework's impact reshapes existing methods of improving LMMs by tapping into diverse global perspectives and engaging contributions. GAP contrasts traditional AI self-assessment methods by ensuring legal and ethical compliance while avoiding issues related to AI-generated data.

Future Directions:

• Development of visually refined LLMs for question generation.
• Enhancing probabilistic models to better estimate model capabilities, controlling for external factors like player skill and image difficulty.
• Extending GAP's scope beyond VQA to address domain-specific challenges in AI.

Conclusion

This paper delineates an innovative approach to fine-tuning LMMs through gamified crowd-sourcing, efficiently addressing gaps in AI models' understanding of visual content. By fostering engagement and harnessing human insights, GAP provides a scalable solution for continuous AI improvement. The promising results anticipate further research in cross-domain applications, ensuring a broad impact on advancing AI capabilities.