MetaSC: Test-Time Safety Specification Optimization for Language Models (2502.07985v2)

Abstract: We propose a novel dynamic safety framework that optimizes LLM (LM) safety reasoning at inference time without modifying model weights. Building on recent advances in self-critique methods, our approach leverages a meta-critique mechanism that iteratively updates safety prompts-termed specifications-to drive the critique and revision process adaptively. This test-time optimization not only improves performance against adversarial jailbreak requests but also in diverse general safety-related tasks, such as avoiding moral harm or pursuing honest responses. Our empirical evaluations across several LLMs demonstrate that dynamically optimized safety prompts yield significantly higher safety scores compared to fixed system prompts and static self-critique defenses. Code released at https://github.com/vicgalle/meta-self-critique.git .

• The paper introduces a novel safety framework that dynamically optimizes safety specifications during inference without modifying model weights.
• It employs a meta-critique mechanism to iteratively update safety prompts, significantly enhancing defenses against adversarial jailbreak attacks.
• Empirical tests on the BiGGen benchmark demonstrate that MetaSC improves safety efficiency and upholds ethical and confidentiality standards.

MetaSC: Optimizing Safety in LLMs at Test Time

The paper, MetaSC: Test-Time Safety Specification Optimization for LLMs, introduces a pioneering safety framework for enhancing the inferential safety of LMs through dynamic optimization without necessitating alterations to model weights. This research leverages the burgeoning advancements in self-critique methodologies, deploying an innovative meta-critique mechanism that iteratively updates and optimizes safety prompts—termed specifications—during inference. The proposed methodology not only heightens resilience against adversarial prompts, or jailbreaks, but also enhances the model's efficiency in general safety tasks, such as maintaining ethical standards and ensuring truthfulness.

Conceptual Foundation and Mechanism

The focal point of the MetaSC framework is the realization that, while embedding safety specifications in pre-training phases provides a foundational layer, substantial improvements can be attained through inference-time computation. This insight is crucial for real-world applications where safety criteria are often fluid and context-dependent. The MetaSC architecture transforms the static approach into a dynamic one by continually refining safety prompts through a meta-critique stage that evaluates and proposes revisions to the specifications based on previous model interactions, thereby maintaining relevance and alignment with evolving safety needs.

Empirical Validation and Results

To substantiate the efficacy of the MetaSC framework, the authors conducted extensive experiments in two main areas: defense against jailbreak prompts and performance in various safety-critical tasks included in the BiGGen benchmark.

1. Defense Against Jailbreak Attacks: The paper employed a set of adversarial prompts and demonstrated that the dynamic optimization of safety specifications through MetaSC significantly increased safety scores across various LMs. This enhancement was consistent across different models, with almost perfect safety measures achieved in some large LMs.
2. General Safety Tasks: Using the BiGGen benchmark, which evaluates multiple dimensions of LLM safety, MetaSC consistently outperformed or matched existing methodologies like static system prompts and static self-critique processes. This was evident in tasks involving ethical sensitivity and maintaining confidentiality, underscoring the system's versatile adaptability to distinct safety challenges.

Theoretical Interpretations and Opportunities

The MetaSC approach is contextualized within the broader framework of inference-time optimization, akin to strategies that enhance reasoning capacities like the LATRO method. Rather than adjusting model weights or relying on external data during training, MetaSC optimizes the discrete variable of safety specifications in an online manner. This reframing of the optimization problem allows for the deployment of LMs that rapidly adjust to new safety contexts, significantly reducing computational demands associated with weight tuning.

Implications and Future Directions

This research advances the understanding of LM safety optimization by offering a promising alternative to traditional weight-based approaches. The ability of MetaSC to enhance model safety without intricate post-training modifications suggests broad applicability across various domains demanding stringent safety standards. Future research could expand on integrating reward-based elements into the meta-critique process to further fine-tune safety responses. Additionally, exploring MetaSC's applicability beyond safety specifications, into domains such as ethical reasoning and security, offers exciting potential.

Conclusion

The introduction of MetaSC marks a transformative step in the domain of LLM safety, presenting a robust framework for the dynamic adaptation of safety measures at the point of inference. Through rigorous empirical validation and a solid theoretical basis, this research not only sets a new benchmark in LM safety but also opens avenues for further exploration and development in AI alignment with human safety and ethical standards.