Mini-Giants: "Small" Language Models and Open Source Win-Win (2307.08189v2)

Abstract: ChatGPT is phenomenal. However, it is prohibitively expensive to train and refine such giant models. Fortunately, small LLMs are flourishing and becoming more and more competent. We call them "mini-giants". We argue that open source community like Kaggle and mini-giants will win-win in many ways, technically, ethically and socially. In this article, we present a brief yet rich background, discuss how to attain small LLMs, present a comparative study of small LLMs and a brief discussion of evaluation methods, discuss the application scenarios where small LLMs are most needed in the real world, and conclude with discussion and outlook.