The Design and Organization of Educational Competitions with Anonymous and Real-Time Leaderboards in Academic and Industrial Settings (2402.07936v1)
Abstract: The goal of this paper is to share our experience in designing and organizing educational competitions with anonymous and (near) real-time leaderboards in both academic and industrial settings. While such competitions serve as a great educational tool and provide participants with hands-on experience, they require significant planning, technical setup, and administration from organizers. In this paper, we first outline several important areas including team registration, data access, submission systems, rules and conditions that organizers should consider when planning such events. We then present a high-level system design that can support (near) real-time evaluation of submissions to power anonymous leaderboards and provide immediate feedback for participants. Finally, we share our experience applying this abstract system in academic and industrial settings. We hope the set of guidelines and the high-level system design proposed here help others in their organization of similar educational events.
- The netflix prize. In Proceedings of KDD cup and workshop, volume 2007, 35. New York, NY, USA.
- Handbook of satisfiability, volume 185. IOS press.
- Bias mitigation in recommender systems to improve diversity. In CIKM, CEUR.
- Codalab. 2021. Codalab. https://codalab.org.
- Understanding and improving the culture of hackathons: Think global hack local. 2015 IEEE Frontiers in Education Conference (FIE), 1–8.
- Deepglobe 2018: A challenge to parse the earth through satellite images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 172–181.
- Gamification in education: A systematic mapping study. Journal of Educational Technology & Society, 18(3): 75–88.
- Dichotomic Pattern Mining Integrated with Constraint Reasoning for Digital Behaviour Analyses. Frontiers in AI.
- The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4): 1–19.
- Stochastic local search: Foundations and applications. Elsevier.
- The international SAT solver competitions. Ai Magazine, 33(1): 89–92.
- Building Higher-Order Abstractions from the Components of Recommender Systems. AAAI.
- Optimized Item Selection to Boost Exploration for RecSys. In CPAIOR.
- Seq2Pat: Sequence-to-pattern generation to bridge mining with machine learning. AI Magazine.
- Kaggle. 2021. Kaggle. https://www.kaggle.com.
- Representing the Unification of Text Featurization using a Context-Free Grammar. AAAI.
- Non-Deterministic Behavior of TS with Linear Payoffs and How to Avoid It. TMLR, 2022.
- Active Learning Meets Optimized Item Selection. In DSO Workshop, IJCAI.
- Cvpr 2020 continual learning in computer vision competition: Approaches, results, current challenges and future directions. arXiv preprint arXiv:2009.09929.
- Surrogate Ground Truth Generation to Enhance Binary Fairness Evaluation in Uplift Modeling. In IEEE ICMLA, 1654–1659.
- Hackathons as an Informal Learning Platform. Proceedings of the 47th ACM Technical Symposium on Computing Science Education.
- Handbook of constraint programming. Elsevier.
- Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3): 211–252.
- Said, A. 2016. A short history of the recsys challenge. AI Magazine, 37(4): 102–104.
- MABWiser: A Parallelizable Contextual MAB Library for Python. In IEEE ICTAI.
- MABWiser: Parallelizable Contextual Multi-armed Bandits. IJAIT, 30(4).
- The 2014 international planning competition: Progress and trends. Ai Magazine, 36(3): 90–98.
- Empowering recommender systems using automatically generated Knowledge Graphs and Reinforcement Learning. CoRR, abs/2307.04996.
- Seq2Pat: Sequence-to-Pattern Generation for Constraint-based Sequential Pattern Mining. In AAAI-IAAI.
- Dichotomic Pattern Mining with Applications to Intent Prediction from Semi-Structured Clickstream Datasets. In KDF-AAAI-22.
- Integer and combinatorial optimization, volume 55. John Wiley & Sons.