LLaMA-Adapter is a lightweight adaptation method designed to efficiently fine-tune LLaMA into an instruction-following model.
The method uses 52K self-instruct demonstrations, introduces 1.2M learnable parameters, and takes less than an hour for fine-tuning on 8 A100 GPUs.
LLaMA-Adapter: A lightweight adaptation method designed to efficiently fine-tune LLaMA into an instruction-following model
52K self-instruct demonstrations: A dataset used for training the LLaMA-Adapter model
1.2M learnable parameters: The number of learnable parameters introduced by LLaMA-Adapter upon the frozen LLaMA 7B model
Zero-init attention mechanism: A proposed mechanism with zero gating that adaptively injects new instructional cues into LLaMA while preserving its pre-trained knowledge
Multi-modal input: An extension of the LLaMA-Adapter approach that allows for image-conditioned LLaMA, improving reasoning capacity on ScienceQA
Pre Trained Knowledge
Instruction Following Model
Multi Modal Input