Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventions (2502.08657v1)

Abstract: Recent AI agents, such as ChatGPT and LLaMA, primarily rely on instruction tuning and reinforcement learning to calibrate the output of LLMs with human intentions, ensuring the outputs are harmless and helpful. Existing methods heavily depend on the manual annotation of high-quality positive samples, while contending with issues such as noisy labels and minimal distinctions between preferred and dispreferred response data. However, readily available toxic samples with clear safety distinctions are often filtered out, removing valuable negative references that could aid LLMs in safety alignment. In response, we propose PT-ALIGN, a novel safety self-alignment approach that minimizes human supervision by automatically refining positive and toxic samples and performing fine-grained dual instruction tuning. Positive samples are harmless responses, while toxic samples deliberately contain extremely harmful content, serving as a new supervisory signals. Specifically, we utilize LLM itself to iteratively generate and refine training instances by only exploring fewer than 50 human annotations. We then employ two losses, i.e., maximum likelihood estimation (MLE) and fine-grained unlikelihood training (UT), to jointly learn to enhance the LLM's safety. The MLE loss encourages an LLM to maximize the generation of harmless content based on positive samples. Conversely, the fine-grained UT loss guides the LLM to minimize the output of harmful words based on negative samples at the token-level, thereby guiding the model to decouple safety from effectiveness, directing it toward safer fine-tuning objectives, and increasing the likelihood of generating helpful and reliable content. Experiments on 9 popular open-source LLMs demonstrate the effectiveness of our PT-ALIGN for safety alignment, while maintaining comparable levels of helpfulness and usefulness.

• The paper introduces PT-ALIGN, a novel method that refines LLM safety with minimal human intervention.
• It employs maximum likelihood estimation and unlikelihood training to iteratively self-refine responses and mitigate harmful outputs.
• Experimental results on nine open-source LLMs show improvements in harmlessness, helpfulness, and honesty, underscoring its scalability and safety impact.

Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventions

In the paper titled "Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventions," the authors propose a novel approach—PT-ALIGN—aimed at enhancing the safety alignment of LLMs, such as ChatGPT and LLaMA, while minimizing human interventions. Traditional methods in aligning LLMs with human intents typically rely heavily on manually annotated data, which is resource-intensive and may lead to biases. PT-ALIGN presents a promising alternative by integrating minimal human annotation with self-refinement processes.

PT-ALIGN leverages the inherent capabilities of LLMs to generate and iteratively refine datasets, focusing on two categories: positive samples, which are expected to represent safe responses, and toxic samples, which deliberately include harmful content to serve as negative reference points. This innovative approach employs fewer than fifty human annotations, significantly reducing the manual workload usually required. By utilizing maximum likelihood estimation (MLE) and fine-grained unlikelihood training (UT), the method guides LLMs toward generating safer content without compromising their helpfulness and effectiveness.

The experimental evaluation spanned nine open-source LLMs, where PT-ALIGN demonstrated notable improvements in safety metrics. The models' performances were thoroughly assessed on multiple fronts, including harmlessness, helpfulness, and honesty. The introduction of extreme toxic samples proved particularly beneficial in strengthening the safety alignment by presenting distinct semantic contrasts, which helped the models effectively differentiate between safe and unsafe content.

The implications of PT-ALIGN are substantial, especially in the context of AI deployment where safety is paramount. By balancing safety with utility, the approach holds promise for broader applications of AI technologies in environments where human safety is critical. Future developments may explore further reducing dependency on human annotations and applying similar methodologies to multimodal models.

Moreover, the scalability of PT-ALIGN highlights its potential adaptability to larger datasets and model sizes, making it a versatile tool in the continuous evolution of safe AI technology. As LLMs become increasingly integrated into various aspects of society, methods like PT-ALIGN that prioritize safety without major sacrifices in utility will be indispensable. The paper ultimately contributes to the discourse on aligning AI behavior with human expectations while optimizing for safety and operational efficiency.