TEGEE: Task dEfinition Guided Expert Ensembling for Generalizable and Few-shot Learning (2403.04233v3)
Abstract: LLMs exhibit the ability to perform in-context learning (ICL), where they acquire new tasks directly from examples provided in demonstrations. This process is thought to operate through an implicit task selection mechanism that involves extracting and processing task definitions from these demonstrations. However, critical questions remain: Which is more essential -- task extraction or definition? And how can these capabilities be further improved? To address these questions, we propose \textbf{TEGEE} (Task Definition Guided Expert Ensembling), a method that explicitly extracts task definitions and generates responses based on specific tasks. Our framework employs a dual 3B model approach, with each model assigned a distinct role: one focuses on task definition extraction, while the other handles learning from demonstrations. This modular approach supports the hypothesis that extracting task definitions is more vital than processing the task itself. Empirical evaluations show that TEGEE performs comparably to the larger LLaMA2-13B model. By leveraging a modular design, our approach extends traditional ICL from few-shot to many-shot learning, supporting an unlimited number of demonstrations and enhancing continual learning capabilities.
- Falcon-40B: an open large language model with state-of-the-art performance.
- Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
- Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
- Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335.
- Making pre-trained language models better few-shot learners. ArXiv, abs/2012.15723.
- Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689.
- Parameter-efficient transfer learning for nlp. arXiv preprint arXiv:1902.00751.
- Lora: Low-rank adaptation of large language models.
- Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269.
- Editing models with task arithmetic. arXiv preprint arXiv:2212.04089.
- Neural network module decomposition and recomposition. arXiv preprint arXiv:2112.13208.
- Openassistant conversations–democratizing large language model alignment. arXiv preprint arXiv:2304.07327.
- Diverse demonstrations improve in-context compositional generalization.
- Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.06259.
- Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470–3487. Association for Computational Linguistics.
- What in-context learning "learns" in-context: Disentangling task recognition and task learning. Annual Meeting of the Association for Computational Linguistics. Accepted to Findings of ACL 2023.
- Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277.
- Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
- Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks.
- Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207.
- Timo Schick and Hinrich Schütze. 2020a. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676.
- Timo Schick and Hinrich Schütze. 2020b. It’s not just size that matters: Small language models are also few-shot learners. Computing Research Repository, arXiv:2009.07118.
- Improving and simplifying pattern exploiting training. ArXiv, abs/2103.11955.
- Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.
- Llama: Open and efficient foundation language models.
- Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. arXiv preprint arXiv:2301.11916.
- Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560.
- Super-naturalinstructions:generalization via declarative instructions on 1600+ tasks. In EMNLP.
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.
- An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080.
- Crossfit: A few-shot learning challenge for cross-task generalization in nlp. arXiv preprint arXiv:2104.08835.
- Contintin: Continual learning from task instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3062–3072. Association for Computational Linguistics.
- Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.
- Composing parameter-efficient modules with arithmetic operations. arXiv preprint arXiv:2306.14870.
- Revisiting few-sample BERT fine-tuning. In International Conference on Learning Representations.