LIMA, a 65B parameter language model, performs well with minimal instruction tuning, demonstrating the importance of pretraining in large language models.
In a controlled human study, LIMA's responses were preferred or equivalent to GPT-4, Bard, and DaVinci003, suggesting limited instruction tuning data is needed for high-quality output.