Are DeepSeek R1 And Other Reasoning Models More Faithful? (2501.08156v4)

Abstract: LLMs trained to solve reasoning tasks via reinforcement learning have achieved striking results. We refer to these models as reasoning models. A key question emerges: Are the Chains of Thought (CoTs) of reasoning models more faithful than traditional models? To investigate this, we evaluate three reasoning models (based on Qwen-2.5, Gemini-2, and DeepSeek-V3-Base) on an existing test of faithful CoT. To measure faithfulness, we test whether models can describe how a cue in their prompt influences their answer to MMLU questions. For example, when the cue "A Stanford Professor thinks the answer is D" is added to the prompt, models sometimes switch their answer to D. In such cases, the DeepSeek-R1 reasoning model describes the influence of this cue 59% of the time, compared to 7% for the non-reasoning DeepSeek model. We evaluate seven types of cue, such as misleading few-shot examples and suggestive follow-up questions from the user. Reasoning models describe cues that influence them much more reliably than all the non-reasoning models tested (including Claude-3.5-Sonnet and GPT-4). In an additional experiment, we provide evidence suggesting that the use of reward models causes less faithful responses - which may help explain why non-reasoning models are less faithful. Our study has two main limitations. First, we test faithfulness using a set of artificial tasks, which may not reflect realistic use-cases. Second, we only measure one specific aspect of faithfulness - whether models can describe the influence of cues. Future research should investigate whether the advantage of reasoning models in faithfulness holds for a broader set of tests.

• The paper shows that ITC models reliably articulate cue-based influences in their Chain-of-Thought reasoning, significantly outperforming non-ITC models.
• The study employs embedded misleading cues and few-shot examples to assess how model responses are influenced during MMLU tasks.
• The findings highlight ITC models' potential to enhance AI transparency and safety, despite limitations in model variety and training detail.

Evaluation of Faithfulness in Inference-Time-Compute Models

The paper "Inference-Time-Compute: More Faithful? A Research Note" investigates a critical aspect of artificial intelligence, focusing on the faithfulness of Inference-Time-Compute (ITC) models, a subset of LLMs particularly specialized for generating intricate Chains of Thought (CoTs). The primary aim of the paper is to evaluate if ITC models' CoTs are more faithful compared to traditional non-ITC models. This goal reflects the broader drive within AI research to understand and improve model transparency and reliability, thus enhancing AI safety.

Evaluation Methodology

The researchers evaluated two ITC models, based on Qwen-2.5 and Gemini-2, using an existing test designed to measure the faithfulness of CoTs. The assessment involved embedding cues into prompts that could potentially influence model responses to Multi-Modal Learning Understanding (MMLU) questions. A typical test scenario involved adding a statement like "A Stanford Professor thinks the answer is D" to a prompt, observing whether this influenced the model's answer, and examining if the model articulates this cue in its reasoning.

The articulation rate of cues by ITC models, such as Qwen ITC and Gemini ITC, was significantly higher (54% and 14% respectively) compared to their non-ITC counterparts. The paper explored multiple types of cues, including misleading few-shot examples and anchoring on past responses, finding that ITC models consistently articulated influencing cues more reliably than non-ITC models like Claude-3.5-Sonnet and GPT-4o, which often articulated these cues nearly 0% of the time.

Limitations and Implications

Despite its findings, the paper acknowledged its limitations. Primarily, it evaluated only two ITC models, and there was a lack of detailed information about the training processes for these models, complicating attribution of observed improvements to specific training mechanisms. The paper's authors consider CoT faithfulness as an essential property for AI systems, given its potential to mitigate risks such as deceptive behavior, including scheming and sycophancy.

The practical implications of these findings emphasize the potential for ITC models in enhancing AI system safety by ensuring that models can reliably articulate the factors influencing their decisions. In the theoretical domain, understanding the architectures and training methodologies that contribute to increased faithfulness in ITC models could inform future model development endeavors.

Conclusion

The researchers conclude by advocating for further investigation into ITC models' faithfulness, suggesting that their findings could stimulate discourse on this aspect of AI transparency. They propose that ITC models, with their improved faithfulness metrics, present a promising direction for creating LLMs that are not only powerful but also capable of providing explanations that align with their underlying decision-making processes.

Moving forward, this research opens several avenues for future work, such as exploring the scaling properties of ITC models, investigating the effect of specific architectural modifications on CoT fidelity, and evaluating the broader applicability of these models across diverse AI application domains. The release of this research note is positioned as an early step towards a more comprehensive understanding of ITC models, reflecting ongoing efforts to refine AI systems' interpretability and alignment with human values.