Prover-Verifier Games improve legibility of LLM outputs (2407.13692v2)

Abstract: One way to increase confidence in the outputs of LLMs is to support them with reasoning that is clear and easy to check -- a property we call legibility. We study legibility in the context of solving grade-school math problems and show that optimizing chain-of-thought solutions only for answer correctness can make them less legible. To mitigate the loss in legibility, we propose a training algorithm inspired by Prover-Verifier Game from Anil et al. (2021). Our algorithm iteratively trains small verifiers to predict solution correctness, "helpful" provers to produce correct solutions that the verifier accepts, and "sneaky" provers to produce incorrect solutions that fool the verifier. We find that the helpful prover's accuracy and the verifier's robustness to adversarial attacks increase over the course of training. Furthermore, we show that legibility training transfers to time-constrained humans tasked with verifying solution correctness. Over course of LLM training human accuracy increases when checking the helpful prover's solutions, and decreases when checking the sneaky prover's solutions. Hence, training for checkability by small verifiers is a plausible technique for increasing output legibility. Our results suggest legibility training against small verifiers as a practical avenue for increasing legibility of large LLMs to humans, and thus could help with alignment of superhuman models.

Enhancing Legibility of LLM Outputs Through Prover-Verifier Games

The paper "Prover-verifier games improve legibility of LLM outputs" presents an innovative method to enhance the trustworthiness and comprehensibility of outputs from LLMs. This research highlights a critical issue: while LLMs can effectively solve problems, their solutions often lack legibility, making it challenging for humans to verify their correctness. The authors propose a novel training algorithm inspired by Prover-Verifier Games to address these concerns, particularly in the domain of grade-school mathematics.

Key Contributions and Methodology

The core contribution of the paper is the introduction of a training framework that balances solution accuracy with legibility through a form of adversarial training involving two main components: helpful provers and sneaky provers, alongside a verifier. This framework is inspired by the Prover-Verifier Games, a concept rooted in computational theory, where one party (prover) must convince another (verifier) of a statement's truthfulness.

Training Dynamics

1. Role of Provers and Verifiers:
   • Helpful Provers: These are tasked with generating correct and legible solutions that the verifier finds convincing.
   • Sneaky Provers: These intentionally create incorrect solutions aiming to deceive the verifier, enhancing its robustness against adversarial attacks.
   • Verifiers: These models are trained to distinguish between valid and invalid solutions, learning from their interactions with provers.
2. Iterative Training Approach:
   • The training proceeds in rounds, dynamically adjusting the role and behavior of provers and verifiers.
   • Throughout the rounds, verifiers are progressively trained on more sophisticated attempts by sneaky provers, improving their discerning ability.
3. Legibility Metrics:
   • Human evaluation is utilized to assess legibility, focusing on the ability of humans to verify solution correctness within time constraints.

Results and Observations

The experimental results demonstrate several critical findings:

• Improvement in Verifier Robustness: The verifiers exhibit increased robustness against sneaky solutions in successive rounds, as evidenced by the decrease in successful adversarial attacks.
• Enhanced Human Legibility: Solutions from helpful provers become more legible to human evaluators over training rounds, as indicated by improved accuracy and reduced evaluation time.
• Legibility-Accuracy Trade-off: The paper highlights a "legibility tax," where optimizing solely for solution correctness reduces legibility, underscoring the need for balanced optimization strategies.

Implications and Future Directions

The approach outlined in this paper offers significant implications for the alignment and oversight of AI systems, particularly as they approach superhuman capabilities. By improving legibility, this method supports human oversight and fosters trust in AI outputs, crucial in high-stakes applications. The research also points to several future directions, such as extending this framework to more complex domains or exploring unsupervised settings where ground truth labels are less available.

In conclusion, the authors lay substantive groundwork for future exploration into scalable oversight methods for AI, where models not only need to be accurate but also need to produce outputs that are transparent and understandable by their human counterparts. The paper compellingly argues that fostering synergy between LLMs and human capabilities through mutual legibility can significantly enhance the alignment and safety of AI systems.