Aligning Large Language Models with Human: A Survey (2307.12966v1)

Abstract: LLMs trained on extensive textual corpora have emerged as leading solutions for a broad array of NLP tasks. Despite their notable performance, these models are prone to certain limitations such as misunderstanding human instructions, generating potentially biased content, or factually incorrect (hallucinated) information. Hence, aligning LLMs with human expectations has become an active area of interest within the research community. This survey presents a comprehensive overview of these alignment technologies, including the following aspects. (1) Data collection: the methods for effectively collecting high-quality instructions for LLM alignment, including the use of NLP benchmarks, human annotations, and leveraging strong LLMs. (2) Training methodologies: a detailed review of the prevailing training methods employed for LLM alignment. Our exploration encompasses Supervised Fine-tuning, both Online and Offline human preference training, along with parameter-efficient training mechanisms. (3) Model Evaluation: the methods for evaluating the effectiveness of these human-aligned LLMs, presenting a multifaceted approach towards their assessment. In conclusion, we collate and distill our findings, shedding light on several promising future research avenues in the field. This survey, therefore, serves as a valuable resource for anyone invested in understanding and advancing the alignment of LLMs to better suit human-oriented tasks and expectations. An associated GitHub link collecting the latest papers is available at https://github.com/GaryYufei/AlignLLMHumanSurvey.

Aligning LLMs with Human: A Survey

The paper comprehensively surveys techniques for aligning LLMs with human expectations, a topic of substantial relevance given the increasing prevalence of LLMs in everyday applications. LLMs, such as GPT-3, have demonstrated impressive capabilities across a range of NLP tasks. However, issues such as misunderstanding instructions, biased content generation, and hallucinations remain challenges. This survey categorizes alignment strategies into three crucial areas: data collection, training methodologies, and evaluation techniques.

Data Collection for Alignment

Data collection strategies are central to LLM alignment. The paper identifies two primary sources: human-provided instructions and those generated by more advanced LLMs. Human-derived data can be sourced from NLP benchmarks and carefully curated instructions. The authors highlight how NLP benchmarks, like FLAN and SuperNaturalInstruction, are employed to adapt existing datasets into instructional formats. While these utilize prompt engineering, their limited scope can restrict real-world applicability. Hand-crafted instructions, addressed through strategies like those employed in OpenAssistant, offer a manual but often richer source of data.

The paper also reflects on "self-instruction" methodologies, where powerful LLMs like GPT-4 are prompted to generate diverse and high-quality instructions, aiming to mitigate the data scarcity issue. Instruction data management explores optimizing the data for LLM training, recognizing that not all instructions contribute equally to LLM capability.

Training Methodologies

Training methodologies discussed include supervised fine-tuning (SFT) and methods incorporating human preferences. Traditional SFT is enhanced by approaches such as reinforcement learning from human feedback (RLHF). The paper discusses the nuances of RLHF and its variations, addressing the prevalent issues such as computational load and training stability. The authors introduce offline strategies which bypass complexities associated with PPO, focusing on ranking-based methods and language-based feedback.

Parameter-efficient training approaches, such as LoRA and QLoRA, present practical solutions to reduce computational demands. These strategies maintain alignment efficacy by only training specific parameters or using quantization techniques.

Evaluation Techniques

Evaluating aligned LLMs requires robust benchmarks. The paper classifies benchmarks into closed-set and open-ended categories. Closed-set evaluations provide a quantifiable measure through predefined answers, while open-ended benchmarks like Vicuna-80 encourage qualitative evaluations by human or LLM-based judges.

In terms of evaluation paradigms, the paper notes that traditional metrics like BLEU or ROUGE are insufficient for open-ended responses, necessitating human or LLM-based evaluations. Research is pointed towards LLMs as evaluators to reduce reliance on costly human annotations, although challenges such as inherent biases in LLM judgments are recognized.

Implications and Future Directions

The paper outlines several forward-looking directions: optimizing instruction data quality, expanding non-English language support, advancing alignment training methodologies, developing human-in-the-loop systems for data generation, and enhancing combined human-LLM evaluation frameworks. These enhancements will likely facilitate the deployment of more robust, culturally sensitive, and user-aligned LLMs.

The thorough analysis in this survey underscores the complexity and importance of aligning LLMs with human expectations. The approaches detailed provide a foundation for understanding how researchers can tackle the multifaceted challenges of LLM alignment, guiding future endeavors in creating more reliable and contextually aware LLMs.