Emma

Summary:

  • LLaMA-Adapter is a lightweight adaptation method designed to efficiently fine-tune LLaMA into an instruction-following model.
  • The method uses 52K self-instruct demonstrations, introduces 1.2M learnable parameters, and takes less than an hour for fine-tuning on 8 A100 GPUs.

Key terms:

  • LLaMA-Adapter: A lightweight adaptation method designed to efficiently fine-tune LLaMA into an instruction-following model
  • 52K self-instruct demonstrations: A dataset used for training the LLaMA-Adapter model
  • 1.2M learnable parameters: The number of learnable parameters introduced by LLaMA-Adapter upon the frozen LLaMA 7B model
  • Zero-init attention mechanism: A proposed mechanism with zero gating that adaptively injects new instructional cues into LLaMA while preserving its pre-trained knowledge
  • Multi-modal input: An extension of the LLaMA-Adapter approach that allows for image-conditioned LLaMA, improving reasoning capacity on ScienceQA

Tags:

Research LLaMA Alpaca ScienceQA LLaMA-Adapter A100 GPUs Pre Trained Knowledge Instruction Following Model 7B parameters Multi Modal Input