OPT: Open Pre-trained Transformer Language Models (2205.01068v4)

Abstract: LLMs, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computational cost, these models are difficult to replicate without significant capital. For the few that are available through APIs, no access is granted to the full model weights, making them difficult to study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which we aim to fully and responsibly share with interested researchers. We show that OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop. We are also releasing our logbook detailing the infrastructure challenges we faced, along with code for experimenting with all of the released models.

Overview of "OPT: Open Pre-trained Transformer LLMs"

The paper "OPT: Open Pre-trained Transformer LLMs" presents a technical report on the development and characteristics of a suite of decoder-only pre-trained transformers, referred to as Open Pre-trained Transformers (OPT), which range from 125 million to 175 billion parameters. The research aims to address the lack of accessibility and transparency regarding LLMs such as GPT-3#1, by openly sharing the models, logbooks, and codebase, ensuring that a broader segment of researchers can engage with these powerful tools.

Key Contributions

1. Model Development and Design: The authors developed eight Transformer LLMs with varying parameter sizes: 125M, 350M, 1.3B, 2.7B, 6.7B, 13B, 30B, 66B, and 175B. These models largely follow the architectures and hyperparameters established by previous works, particularly GPT-3, but incorporate contemporary best practices in data collection and training efficiency.
2. Resource Sharing: Complete weights for OPT models up to 66B are freely available, with OPT-175B accessible upon request for research purposes. This makes them unique among very LLMs, most of which are either proprietary or accessible only via paid APIs.
3. Efficiency: OPT-175B was trained using a significantly lower carbon footprint—only 1/7th of GPT-3#1's training emissions—thanks to the use of modern NVIDIA hardware and optimized training techniques.
4. Performance: The performance of OPT-175B was benchmarked against GPT-3 and other LLMs across various standard NLP and dialogue tasks. While showing similar performance across standard metrics, certain discrepancies were noted in specific tasks and safety benchmarks.
5. Responsibility and Ethical Release: The authors emphasize the importance of responsible AI and ethical considerations. They highlight potential risks associated with LLMs, such as bias and toxicity, and provide detailed evaluations to identify and understand these risks.

Numerical Results

On standard NLP tasks, OPT-175B demonstrates competitiveness with GPT-3. For instance, zero-shot performance average across 14 tasks aligns closely with GPT-3, although individual task performance varied. Specific results include:

• Zero-shot Accuracy: Approximate parity with GPT-3 on 10 tasks, with minor underperformance on MultiRC and overperformance on WIC.
• Training Efficiency: OPT-175B achieved up to 147 TFLOP/s per 80GB A100 GPU, demonstrating significant computational efficiency.
• Carbon Footprint: The development of OPT-175B resulted in approximately 75 tons of CO2eq emissions versus the 500 tons estimated for GPT-3#1.

Bias and Safety Evaluations

Legitimate concerns regarding biases and toxic content generation are addressed through comprehensive evaluations:

• Hate Speech Detection: OPT-175B performs better than GPT-3 Davinci, particularly in one-shot and few-shot binary/multiclass settings.
• CrowS-Pairs: OPT-175B exhibits more stereotypical bias overall compared to Davinci, likely due to the prevalence of unmoderated social media text in the training data.
• StereoSet: Shows comparable performance to Davinci in idealized context association test scores (ICAT), although nuances exist across bias categories.
• RealToxicityPrompts: Higher propensity for toxic outputs compared to PaLM and Davinci, especially as prompt toxicity increases.

Implications and Future Directions

The implications of this work extend to both practical implementations and theoretical advancements in AI. Practically, the openly available OPT models foster reproducibility and inclusivity in AI research, allowing more diverse contributions to the paper and mitigation of LLM-related risks. The theoretically significant reduction in training emissions presents a more sustainable approach to developing large models.

Future developments may focus on issues of factual accuracy, bias mitigation, and the use of retrieval-augmented techniques to enhance model performance on fact-dependent tasks. Additionally, further investigation into the capabilities and limitations of these models across unexplored or less-studied languages and domains is warranted.

Conclusion

The release of OPT marks a significant step towards transparency and collaboration within the AI research community. By openly sharing extensive model details and computational logs, the authors aim to democratize access to cutting-edge LLMs, promoting a balanced dialogue on their societal impacts while enabling new avenues for innovation and ethical AI development.