Can LLMs Follow Simple Rules? (2311.04235v3)

Abstract: As LLMs are deployed with increasing real-world responsibilities, it is important to be able to specify and constrain the behavior of these systems in a reliable manner. Model developers may wish to set explicit rules for the model, such as "do not generate abusive content", but these may be circumvented by jailbreaking techniques. Existing evaluations of adversarial attacks and defenses on LLMs generally require either expensive manual review or unreliable heuristic checks. To address this issue, we propose Rule-following Language Evaluation Scenarios (RuLES), a programmatic framework for measuring rule-following ability in LLMs. RuLES consists of 14 simple text scenarios in which the model is instructed to obey various rules while interacting with the user. Each scenario has a programmatic evaluation function to determine whether the model has broken any rules in a conversation. Our evaluations of proprietary and open models show that almost all current models struggle to follow scenario rules, even on straightforward test cases. We also demonstrate that simple optimization attacks suffice to significantly increase failure rates on test cases. We conclude by exploring two potential avenues for improvement: test-time steering and supervised fine-tuning.

• The paper presents the RuLES framework to programmatically measure LLMs’ adherence to predefined 'harmless' and 'helpful' rules.
• It finds that even advanced models like GPT-4 struggle with rule-following, especially for 'helpful' directives under adversarial conditions.
• The results highlight the need for novel model architectures and fine-tuning methods to enhance rule compliance for safe, real-world deployments.

Evaluation of Rule-Adherence in LLMs with RuLES

The research paper "Can LLMs Follow Simple Rules?" introduces the Rule-following Language Evaluation Scenarios (RuLES) framework, a methodological approach to test and measure the capability of LLMs in adhering to pre-defined rules. This investigation is especially significant in the context of deploying LLMs for real-world applications, where reliability and safety are paramount. The paper proposes RuLES as a programmatic paradigm to quantify the extent to which LLMs can be constrained by explicit behavior prescriptions, thus addressing limitations in existing adversarial evaluation efforts that depend heavily on manual oversight or heuristic evaluations.

Core Contributions

The paper presents several key contributions in the form of RuLES, a framework comprising 14 well-defined scenarios with associated rules which LLMs must follow. In these scenarios, models are provided with clear instructions that are straightforwardly evaluated through programmatic means, thus enhancing the objectivity and repeatability of the evaluation process. Each scenario challenges the model with specific instructions and employs evaluation functions to assess whether the LLM responds in accordance with the prescribed rules.

The RuLES framework classifies rules into two categories: 'harmless', which define actions the model should refrain from, and 'helpful', which mandate positive actions the model should undertake. This dual classification parallels safety and liveness properties in traditional computing systems, thereby integrating principles of software reliability into the assessment of AI models. Furthermore, RuLES is recognized for its capacity to autonomously verify rule adherence through automated checks, circumventing the need for costly human evaluations.

Evaluation and Findings

The evaluations elucidated in the paper reveal that current LLMs, both open-source and proprietary, struggle notably in scenarios that require strict rule adherence. For instance, leading models like OpenAI's GPT-4 perform relatively well but are not immune to failure, especially when faced with adversarial manipulations. The evaluation also highlights the disparity in adherence between different rule types, with models generally finding 'helpful' rules more challenging than 'harmless' ones.

The authors also explore the effect of system messages versus user messages in guiding model behavior. Results indicate that the presentation of rules in system messages offers marginal gains, suggesting that the fundamental challenge lies in the inherent capacities of the models rather than the format of rule presentation.

Practical and Theoretical Implications

The research underscores the critical need for developing LLMs that can better adhere to specified rules, an essential requirement for their safe application in environments demanding ethical and legal compliance. The inability to reliably follow rules poses significant risks, especially when models are tasked with sensitive or high-stakes roles.

On a theoretical level, the paper suggests that advancements in alignment techniques need diversification beyond current methodologies which, while addressing harmful outputs, may not sufficiently capture nuanced rule-following requirements. The identified gap between existing alignment-focused models and the necessary rule-adherence capability highlights an opportunity for further research into novel model architectures and training paradigms specifically targeting rule compliance.

Future Directions

The authors identify promising avenues for future research, notably fine-tuning mechanisms and the introduction of dynamic steering methods to improve rule adherence dynamically during inference. Experimentation with supervised fine-tuning on easy rule scenarios showed potential for significant improvements, even indicating transferability to more adverse scenarios. Another proposed approach involves real-time adjustments to model outputs, employing methodologies inspired by reinforcement learning and ensemble strategies.

In summary, the paper offers a comprehensive framework for evaluating the rule-following capabilities of LLMs, highlighting current limitations and suggesting practical pathways to enhance model reliability. RuLES emerges as a pivotal tool in bridging the gap between theoretical alignment considerations and practical, reliable AI behaviors essential for trustworthy deployment in real-world applications.