Instruction Fine-Tuning: Does Prompt Loss Matter?

Abstract: We present a novel study analyzing the effects of various prompt loss token weights (PLW) for supervised instruction fine-tuning (SIFT). While prompt-masking (PLW = 0) is common for SIFT, some fine-tuning APIs support fractional PLWs and suggest that using a small non-zero PLW can help stabilize learning when fine-tuning on short-completion data. However, there has never been a study confirming this claim, and OpenAI, a major cloud-based SIFT provider, recently removed this parameter from their fine-tuning API. We found that performance of models fine-tuned on short-completion data had a statistically-significant negative quadratic relationship with PLW. Using small values (0.01 - 0.5) of PLW produced better results on multiple-choice and short-generation benchmarks (outperforming models fine-tuned on long-completion data) while large values (~ 1.0) of PLW produced better results on long-generation benchmarks. We explained this effect and verified its importance through additional experiments. This research serves as a warning to API providers about the importance of providing a PLW parameter for SIFT.

• The paper finds that prompt loss weighting has a statistically significant, negatively quadratic effect on downstream performance for short-completion datasets.
• The study employs a detailed methodology by reproducing Stanford’s Alpaca experiment with diverse PLW settings, PLMs, and extensive benchmark evaluations.
• The results imply that fine-tuning can be simplified for long completions while optimizing PLW for short completions enhances overall model accuracy.

Introduction

Advances in language modeling have catalyzed a steep increase in the explorations and applications within the field. Despite the growth and evolution of state-of-the-art performance, the frameworks and best practices are continuously emerging. One such practice involves the fine-tuning of pre-trained LLMs (PLMs) leveraging prompt token classification loss weighting (PLW). This research explores the effects of PLW on LLMs, using the case study of 7B-sized LLaMA models. Specifically, the investigation centers on the impact of PLW on models fine-tuned on instruction tasks, revealing intriguing findings about the relationship between PLW, data set characteristics, and downstream task performance.

Motivation and Hypotheses

The primary focus here is the instruction-fine tuning of LLMs, wherein a model is trained to respond accurately to a given prompt. The dichotomy between short-completion and long-completion data underpins the premise of this study. The former has average completion length shorter than the average prompt length, while the latter has longer completions. Conventional belief, as stated by OpenAI, posits a beneficial role of non-zero PLW, particularly in short-completion contexts where it could potentially stabilize training. However, there is no consensus or compelling evidence to precisely quantify the impact. The study aims to clarify the relationship between PLW and model performance—whether it be quadratic, as hypothesized, and if so, to what extent it affects short versus long-completion data sets.

Methodology

The research incorporates an elaborate methodology involving the reproduction of Stanford's Alpaca experiment using diverse PLW settings, PLMs, and fine-tuning datasets. The complexity is seen in the selection of loss weights, ranging from zero (akin to the masking strategy) to one (unmasked training as per the default of the Transformer library). This allows for a comprehensive examination across the spectrum of potential PLW influences. The models' performance was assessed on eight benchmarks, covering a range of multi-choice and generative tasks from translation to short and medium-length completions.

Results and Discussion

In a remarkable affirmation of the study's hypotheses, the results indicated that PLW exhibits a statistically significant, negatively quadratic relationship with downstream performance for models trained on short-completion datasets. In contrast, models trained on long-completion datasets did not display a significant PLW impact, suggesting an element of robustness irrespective of the PLW parameter. These outcomes suggest the possible dispensability of prompt loss weighting and masking for long-completion data fine-tuning—thereby offering a potential simplification of the instruction fine-tuning process. Moreover, by pinpointing an optimal PLW for short-completion scenarios, the research contributes a nuanced perspective on how fine-tuning can be calibrated to harness the full potential of LLMs within specific data contexts.

The implications of this study are dual-faceted. For one, it supports the experimental optimization of PLW parameters tailored to the data set at hand. Secondly, the research pushes the boundaries of LLM fine-tuning best practices, advocating for data-specific personalization over a one-size-fits-all approach. The conclusive evidence pertaining to PLW reaffirms the necessity for continuous and deliberate experimentation in the ongoing refinement of LLM instructional fine-tuning strategies.