Usefulness of Human-Written Internal Thoughts for LLMs

Determine whether datasets of human-written internal thought processes are useful for large language models when used to train a judge model to evaluate internal thoughts, including whether such human thought data provides benefits comparable to the needs of large language models’ thought generation and response improvement.

Background

In optimizing their Thought Preference Optimization pipeline, the authors deliberately avoid feeding the thought sequences to a judge model and instead evaluate only final responses. They note a lack of judge models capable of assessing internal thoughts and the difficulty of collecting human thought data.

Crucially, they state uncertainty about the value of human-written thoughts for LLMs even if such data were collected. Resolving this uncertainty would clarify whether training a judge (or models) on human thought annotations is beneficial for LLMs’ internal reasoning and thus inform future training strategies and data collection efforts.