Diversified Ensembling: An Experiment in Crowdsourced Machine Learning (2402.10795v1)

Abstract: Crowdsourced machine learning on competition platforms such as Kaggle is a popular and often effective method for generating accurate models. Typically, teams vie for the most accurate model, as measured by overall error on a holdout set, and it is common towards the end of such competitions for teams at the top of the leaderboard to ensemble or average their models outside the platform mechanism to get the final, best global model. In arXiv:2201.10408, the authors developed an alternative crowdsourcing framework in the context of fair machine learning, in order to integrate community feedback into models when subgroup unfairness is present and identifiable. There, unlike in classical crowdsourced ML, participants deliberately specialize their efforts by working on subproblems, such as demographic subgroups in the service of fairness. Here, we take a broader perspective on this work: we note that within this framework, participants may both specialize in the service of fairness and simply to cater to their particular expertise (e.g., focusing on identifying bird species in an image classification task). Unlike traditional crowdsourcing, this allows for the diversification of participants' efforts and may provide a participation mechanism to a larger range of individuals (e.g. a machine learning novice who has insight into a specific fairness concern). We present the first medium-scale experimental evaluation of this framework, with 46 participating teams attempting to generate models to predict income from American Community Survey data. We provide an empirical analysis of teams' approaches, and discuss the novel system architecture we developed. From here, we give concrete guidance for how best to deploy such a framework.

• The paper shows that an ensembled global model outperforms individual submissions by integrating specialized subgroup improvements.
• It highlights how teams used domain expertise to focus on specific model weaknesses, enhancing fairness and performance.
• The study introduces a novel algorithmic framework and system design that democratizes model development and guides future ML competitions.

Diversifying Approaches in Crowdsourced Machine Learning through Specialization and Ensembling

Introduction

The proliferation of machine learning competitions on platforms like Kaggle represents a significant shift in how predictive models are developed, often harnessing the collective expertise of a global community. However, the traditional competitive framework, where the singular best model wins, may not optimally utilize the diverse strengths of all participants. A paper titled "Diversified Ensembling: An Experiment in Crowdsourced Machine Learning" examines an alternative crowdsourcing model that encourages participants to specialize and focuses on ensemble techniques to integrate these specialized models into a cohesive and potentially superior global model. This approach not only democratizes participation by leveraging unique insights from a broader spectrum of individuals but also addresses fairness concerns by enabling competitors to target subgroup improvements directly.

Experimental Setup and Key Findings

The paper outlines an empirical evaluation of this novel crowdsourcing framework through a medium-scale experiment involving 46 teams attempting to predict income based on demographic data. The competition diverged from the standard model by allowing teams to submit model-group pairs, with the idea that these pairs could be ensembled into a global model that benefits from specialized local improvements.

The key outcomes underscore the potential of this diversified ensemble approach:

• Performance Gains: The ensembled global model outperformed all individual models, suggesting that the whole is greater than the sum of its parts in machine learning competitions.
• Specialization Benefits: Teams specialized using both algorithmic methods and domain insight, addressing specific subgroups where they suspected or identified model underperformance.
• Quality Insights Through Diversity: By diversifying the types of improvements sought (across subgroups), the competition unearthed a richer set of insights into the model performance, particularly regarding fairness.

Theoretical and System Contributions

Beyond the empirical findings, this work offers several contributions to the field of machine learning and crowdsourcing:

• Algorithmic Framework: It validates the theoretical algorithmic framework put forth by previous research, emphasizing the importance of specialization and ensemble learning in improving model performance and fairness.
• System Design: A novel system architecture capable of handling this new competition format is described, providing a blueprint for future implementations.
• Practical Guidance: The paper offers concrete guidance for deploying such a framework effectively, informing both academic experiments and industry applications.

Implications and Future Directions

This research has notable implications for both the theory and practice of machine learning:

• Broader Participation: By enabling specialization, this framework can engage a wider community in model development, including individuals with unique domain knowledge but perhaps limited machine learning expertise.
• Focus on Fairness: The ability to target subgroup improvements directly provides a structured way to address fairness, a growing concern in AI applications.
• Methodological Innovation: The approach encourages methodological innovation, as participants experiment with both manual and automated techniques to identify subgroups and improve model performance in these areas.

Looking ahead, the paper also highlights areas for future exploration, including adapting the framework for more complex prediction tasks and further refining the system design for scalability and security.

Conclusion

The "Diversified Ensembling" experiment provides a compelling case for rethinking how machine learning competitions are structured, offering a more inclusive and potentially more effective model for crowdsourced AI development. By harnessing the collective strengths of a diverse set of participants and emphasizing fairness through direct subgroup targeting, this approach represents a meaningful step towards democratizing machine learning and addressing some of its most pressing ethical challenges.