Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models (2410.02205v3)

Abstract: LLMs are expected to be predictable and trustworthy to support reliable decision-making systems. Yet current LLMs often show inconsistencies in their judgments. In this work, we examine logical preference consistency as a foundational requirement for building more dependable LLM systems, ensuring stable and coherent decision-making while minimizing erratic or contradictory outputs. To quantify the logical preference consistency, we propose a universal evaluation framework based on three fundamental properties: transitivity, commutativity and negation invariance. Through extensive experimentation across diverse LLMs, we demonstrate that these properties serve as strong indicators of judgment robustness. Furthermore, we introduce a data refinement and augmentation technique, REPAIR, that enhances logical consistency while maintaining alignment with human preferences. Finally, we show that improving consistency leads to better performance in LLM-driven logic-based algorithms, reinforcing stability and coherence in decision-making systems.

• The paper introduces a novel evaluation framework that uses transitivity, commutativity, and negation invariance to quantify logical consistency in large language models.
• It reveals that logical performance varies across LLMs and that targeted data refinement and augmentation improve model coherence.
• The study highlights that enhanced logical consistency leads to more reliable outcomes in downstream tasks such as pairwise-preference search.

Analyzing Logical Consistency in LLMs

The paper "Aligning with Logic: Measuring, Evaluating and Improving Logical Consistency in LLMs" explores the nuanced examination of logical consistency in LLMs, a feature critical for ensuring the reliability and predictability of AI systems. The research highlights the persistent issue of inconsistency in LLMs' decision-making and asserts logical consistency as an essential factor for trustworthy AI systems.

Logical consistency is crucial, as it underpins stable and coherent decision-making processes, minimizing erratic or contradictory outputs that can undermine AI reliability. To quantitatively assess this consistency, the authors propose a universal framework that employs three logical proxies: transitivity, commutativity, and negation invariance. This framework provides a means to evaluate and improve logical consistency, offering a metric for robustness across various LLMs.

Framework and Evaluation

The framework for logical consistency evaluates transitivity, commutativity, and negation invariance through systematic tests. Transitivity ensures that if a model prefers A to B and B to C, it must also prefer A to C, thus maintaining internal coherence. Commutativity examines invariance in the outcome when the order of items is swapped, testing whether models are susceptible to permutation bias. Negation invariance evaluates the model's ability to handle inverse relationships, maintaining logical integrity when statements are negated.

The results demonstrate that logical consistency varies significantly across different LLMs and tasks. For instance, recent models like Gemma-2-9B and Phi-3-medium illustrate stronger logical consistency compared to earlier versions, indicating advancements in model training and fine-tuning. However, it’s noted that while some LLMs excel in one logical aspect, they may perform poorly in others, highlighting the complexity of achieving comprehensive logical consistency.

Data Refinement and Augmentation

Addressing the inconsistency challenge, the research introduces a novel data refinement and augmentation method to enhance logical consistency during training without sacrificing alignment with human preferences. The process involves refining noisy datasets to establish coherent rankings via win-loss rates and subsequently augmenting them with logically extrapolated comparisons. This method has been shown to improve logical consistency, highlighting the importance of clean and logically consistent training data.

Implications for Downstream Applications

The impact of logical consistency extends to downstream applications, where LLMs are employed as logical operators. The paper explores this in the context of the Pairwise-Preference Search (PairS) algorithm, a sorting-based method reliant on logical consistency. LLMs with higher logical consistency yield better performance in such applications, underscoring the necessity of logical coherence in AI systems employed in decision-making processes.

Conclusion

The authors argue for the integration of logical consistency as a key component in developing trustworthy AI systems. The findings indicate that logical consistency is a strong proxy for model reliability and that improvements in data handling and model training can significantly enhance this trait. The research invites further exploration into logical consistency to underpin more reliable and effective LLM-based applications, serving as a cornerstone for advancing AI dependability and utility in complex decision-making environments.