Thought-Centric Preference Optimization

• TCPO is a preference learning approach that applies supervision to intermediary reasoning artifacts such as thought trees and stepwise chains rather than only final responses.
• Key contributions include novel optimization objectives and credit assignment methods that enhance decision-making accuracy and efficiency across various domains.
• The approach supports diverse modalities by integrating textual, visual, and embodied reasoning frameworks to drive improved performance.

Thought-Centric Preference Optimization (TCPO) denotes a class of preference-learning methods in which supervision is attached to explicit reasoning artifacts—such as thought-tree paths, chain-of-thought steps, rationale summaries, hidden thought prefixes, reasoning topologies, visual region-wise chains, or stepwise thought–action segments—rather than only to final answers. Across recent work, the term is used explicitly for embodied decision-making, while closely related methods instantiate the same pattern under names such as preference optimization on thought trees, reasoning traces, multi-branch preference trees, visual CoT, long-versus-short reasoning trajectories, and hidden thought–answer trajectories (Jiao et al., 10 Sep 2025, Li et al., 2024, Zhang et al., 2024, Lahlou et al., 2024, Liao et al., 2024, Wu et al., 2024, Yang et al., 17 Feb 2025).

1. Conceptual emergence and representative forms

The central shift in TCPO is that the preference unit is a reasoning-bearing object. In conventional response-level preference learning, a prompt is paired with preferred and dispreferred final responses. In TCPO-style systems, the preferred object may instead be a thought-tree path summarized as a rationale, a next reasoning sentence conditioned on a partial chain, a ranked list of search trajectories, a hidden “think-then-answer” sequence, a visual box-selection step plus region-conditioned answer, or a thought-conditioned action decision (Li et al., 2024, Zhang et al., 2024, Liao et al., 2024, Wu et al., 2024, Zhao et al., 25 Apr 2025, Jiao et al., 10 Sep 2025).

Paper	Thought object	Preference source
"Calibrating LLMs with Preference Optimization on Thought Trees for Generating Rationale in Science Question Scoring" (Li et al., 2024)	Thought-tree path summarized as a rationale	Correct rubric-aligned path vs incorrect path
"Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs" (Zhang et al., 2024)	Step-level CoT thought	ToT-selected child vs sibling thoughts
"PORT: Preference Optimization on Reasoning Traces" (Lahlou et al., 2024)	Next reasoning-step sentence	Correct step vs corrupted or weak-LLM step
"TPO: Aligning LLMs with Multi-branch & Multi-step Preference Trees" (Liao et al., 2024)	Ranked trajectory list from a preference tree	Graded trajectory scores and listwise ranking
"Thinking LLMs: General Instruction Following with Thought Generation" (Wu et al., 2024)	Hidden thought–answer trajectory	Judge score on answer only
"Unsupervised Visual CoT Reasoning via Preference Optimization" (Zhao et al., 25 Apr 2025)	Region-wise visual thought chain	Evaluator ranking of box-conditioned chains
"RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for LLMs" (Yan et al., 3 Jun 2025)	CoT-plus-answer tuple	Retrieval-grounded preferred vs weaker rejected trajectory
"TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-making" (Jiao et al., 10 Sep 2025)	Stepwise {"thoughts", "action"} segment	Higher-return trajectory segment vs lower-return segment

Later variants extend the same pattern in two additional directions. "Thinking Preference Optimization" trains on long CoT responses as chosen answers and short CoT responses as rejected answers for the same question, making the preferred object an entire reasoning trajectory rather than a single answer string (Yang et al., 17 Feb 2025). "TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization" represents reasoning as a graph $G=(V,E)$ of sub-claims and support relations, and shapes the preference margin with topology quality and uncertainty, so the optimized object is a response together with its reasoning topology rather than a flat completion (Abdullah et al., 30 Apr 2026).

2. Representations of “thought”

TCPO systems differ chiefly in how they make reasoning explicit. In thought-tree scoring, the task is decomposed over key answer elements $K=\{k_j\}_{j=1}^M$, each node stores a binary Yes/No assessment decision about whether a student response satisfies $k_j$, and each root-to-leaf path yields a score via a rubric function $f_r$. The tree is an LLM-generated Monte Carlo approximation to human KAE-level decisions, and a path has probability $P(\text{path}_l)=\prod_{j=1}^M P(z_j)$ and score $\hat y_{\text{path}_l}=f_r(\text{path}_l)$ (Li et al., 2024). This makes the “thought” a discrete, rubric-grounded trajectory.

Search-based reasoning papers instead encode thought as a sequence of natural-language steps. In CPO, a state is $s_{i-1}=[x,z_1,\dots,z_{i-1}]$, candidate next thoughts $z_i^{(j)}$ are generated from that state, and ToT search identifies preferred children. In PORT, a chain-of-thought is a sequence of reasoning sentences $z^1,\dots,z^n$, and the preferred unit is the correct next sentence $z^k$ conditioned on the prefix $K=\{k_j\}_{j=1}^M$0 (Zhang et al., 2024, Lahlou et al., 2024). TPO generalizes this from a single preferred chain to a list $K=\{k_j\}_{j=1}^M$1 of trajectories, each a sequence of steps, with a score vector $K=\{k_j\}_{j=1}^M$2 derived from a preference tree (Liao et al., 2024).

Several systems use latent or summarized thoughts rather than exposed step labels. In "Thinking LLMs," the model outputs a thought prefix $K=\{k_j\}_{j=1}^M$3 and a response $K=\{k_j\}_{j=1}^M$4, but only $K=\{k_j\}_{j=1}^M$5 is shown to the judge; the optimized sequence is the joint trajectory $K=\{k_j\}_{j=1}^M$6 (Wu et al., 2024). In science scoring, the rationale is a response-level summary tightly coupled to a specific thought-tree path, and in RACE-Align each response is a tuple $K=\{k_j\}_{j=1}^M$7, formatted as > reasoning </think> answer, so the full preference object is a CoT-plus-answer trajectory (Li et al., 2024, Yan et al., 3 Jun 2025).

Multimodal TCPO broadens the thought substrate beyond text. UV-CoT defines a visual thought as a region-conditioned reasoning step: a bounding box together with a textual response conditioned on the cropped region, producing a response chain $K=\{k_j\}_{j=1}^M$8 over multiple visual inspection steps (Zhao et al., 25 Apr 2025). TUR-DPO replaces linear chains by small reasoning topologies $K=\{k_j\}_{j=1}^M$9, where nodes are atomic sub-claims and edges represent support or dependency; topology features such as minimal valid path coverage, cycle count, dangling nodes, and contradiction score are combined into a topology score $k_j$0 (Abdullah et al., 30 Apr 2026). In embodied TCPO, the thought is operationalized as a JSON object with a textual thought $k_j$1 and a discrete action $k_j$2, making the optimized state a thought–action segment inside an interactive trajectory (Jiao et al., 10 Sep 2025).

3. Constructing thought-centric preference data

A defining feature of TCPO is that preference labels are often synthesized from structure, rules, or evaluators rather than directly annotated by humans. In science scoring, a path whose score matches the gold mark is treated as correct, its summarized rationale becomes $k_j$3, and any path with mismatched score yields $k_j$4, inducing preference pairs $k_j$5 over full rationales tied to thought-tree paths (Li et al., 2024). This is a correctness-derived rationale preference dataset over reasoning trajectories.

Search-based variants extract preferences from branching structure. CPO runs ToT search, treats the thought on the selected path as preferred, and treats non-path siblings generated from the same parent state as dispreferred. PORT constructs step-level preferences by pairing the correct next reasoning sentence with a rejected step generated either by digit corruption or by weak-LLM prompting, so the negative sample is a nearby but wrong thought rather than an unrelated answer (Zhang et al., 2024, Lahlou et al., 2024). TPO moves beyond binary sampling by keeping all trajectories in a preference tree and using a ranked list with graded rewards, so even partially correct losers contribute relational information such as $k_j$6 among dispreferred chains (Liao et al., 2024).

Other TCPO systems derive preferences from external grounding or answer-only judgment. RACE-Align generates rejected samples with weaker utilization of external references and more direct reasoning, then generates preferred trajectories by integrating $k_j$7 and TCM diagnostic logic, so the preference is partly about how retrieved knowledge is used inside the CoT (Yan et al., 3 Jun 2025). "Thinking LLMs" samples multiple hidden-thought trajectories for the same instruction, scores only the response part with ArmoRM or STE, optionally adjusts scores by a length-control term $k_j$8, and then labels the best and worst full trajectories as chosen and rejected (Wu et al., 2024). ThinkPO uses existing long CoT responses as chosen answers and short CoT responses as rejected answers for the same question, turning long-versus-short reasoning style into an offline preference dataset without collecting new long traces (Yang et al., 17 Feb 2025).

The same principle extends to multimodal and embodied settings. UV-CoT samples multiple candidate visual thoughts at each timestep, scores each candidate by current-step quality and expected next-step utility, $k_j$9, and forms winner–loser pairs of response chains that differ in the visual thought at step $f_r$0 (Zhao et al., 25 Apr 2025). Embodied TCPO converts sparse environment feedback into dense stepwise preferences: for ALFWorld, the preference score is $f_r$1, and trajectory segments with higher score are preferred over lower-scoring segments that share the same prefix (Jiao et al., 10 Sep 2025). This suggests that, within TCPO, preference construction is often the primary modeling decision.

4. Optimization objectives and credit assignment

Most TCPO methods retain a DPO-style core and change the semantic meaning of the compared object. In its standard form,

$f_r$2

In TCPO, $f_r$3 may denote a rationale summarizing a thought-tree path, a next reasoning sentence, a full <think> reasoning answer trajectory, or a long CoT response. Science scoring applies DPO to full rationales after SFT on synthetic rationales; PORT applies DPO to reasoning-step sentences; ThinkPO applies DPO to long-versus-short CoT sequences; and "Thinking LLMs" applies DPO to hidden thought–answer trajectories whose reward depends only on the visible answer (Li et al., 2024, Lahlou et al., 2024, Yang et al., 17 Feb 2025, Wu et al., 2024).

The main departures from vanilla DPO address richer thought structure and finer credit assignment. CPO argues that full-path preference optimization suffers from longest-common-prefix gradient cancellation when preferred and dispreferred paths share large prefixes, and therefore uses step-level DPO on sibling thoughts instead of full-path DPO (Zhang et al., 2024). TPO replaces binary winner–loser training by a Preference List Ranking loss over all ordered pairs with $f_r$4,

$f_r$5

and further uses Adaptive Step Reward to concentrate margin on discriminative reasoning steps rather than shared prefixes (Liao et al., 2024). UV-CoT introduces Score-DPO, which subtracts a score-dependent margin $f_r$6 inside the sigmoid so that larger evaluator score gaps induce stronger preference pressure (Zhao et al., 25 Apr 2025).

Some variants modify the preference margin with structured side signals. RACE-Align keeps standard DPO but defines the optimized sequence to include both CoT and answer, so the log-probability difference already reflects preference over the reasoning process itself (Yan et al., 3 Jun 2025). TUR-DPO augments DPO with a factorized reward $f_r$7 over semantic faithfulness, topology quality, and uncertainty, and multiplies the loss by a weight $f_r$8 derived from uncertainty, yielding an uncertainty-weighted, topology-aware DPO objective (Abdullah et al., 30 Apr 2026). Embodied TCPO makes thought tokens the primary target and adds two further terms: Action Probability Weighting (APW), which weights thought updates by $f_r$9, and Action Policy Consistency (APC), an L2 penalty that keeps $P(\text{path}_l)=\prod_{j=1}^M P(z_j)$0 close to $P(\text{path}_l)=\prod_{j=1}^M P(z_j)$1 (Jiao et al., 10 Sep 2025). A plausible implication is that “TCPO” refers less to a single loss than to a family of preference objectives whose compared items explicitly encode intermediate reasoning.

5. Empirical performance across domains

In text reasoning and mathematical problem solving, TCPO-style methods repeatedly outperform answer-only or positive-only baselines. CPO improves accuracy by about $P(\text{path}_l)=\prod_{j=1}^M P(z_j)$2 percentage points over vanilla CoT and about $P(\text{path}_l)=\prod_{j=1}^M P(z_j)$3 points over TS-SFT on average across seven datasets, while remaining about $P(\text{path}_l)=\prod_{j=1}^M P(z_j)$4 faster than ToT at inference because search cost is moved into training (Zhang et al., 2024). PORT reports up to relative $P(\text{path}_l)=\prod_{j=1}^M P(z_j)$5 and $P(\text{path}_l)=\prod_{j=1}^M P(z_j)$6 increases in accuracy on GSM8K and AQuA respectively, without extra annotations, by preferring correct next reasoning steps over corrupted or weak-LLM steps (Lahlou et al., 2024). TPO consistently outperforms DPO across five public LLMs on four math datasets; for example, Qwen2-7B-Instruct rises from $P(\text{path}_l)=\prod_{j=1}^M P(z_j)$7 to $P(\text{path}_l)=\prod_{j=1}^M P(z_j)$8 with DPO and to $P(\text{path}_l)=\prod_{j=1}^M P(z_j)$9 with TPO, while DeepSeekMath-7B-Instruct rises from $\hat y_{\text{path}_l}=f_r(\text{path}_l)$0 to $\hat y_{\text{path}_l}=f_r(\text{path}_l)$1 with DPO and to $\hat y_{\text{path}_l}=f_r(\text{path}_l)$2 with TPO (Liao et al., 2024). ThinkPO further improves post-SFT long-CoT reasoning without new long traces: on a reproduced Qwen-2.5-7B setup, average accuracy rises from $\hat y_{\text{path}_l}=f_r(\text{path}_l)$3 to $\hat y_{\text{path}_l}=f_r(\text{path}_l)$4 over five benchmarks while average output length rises from $\hat y_{\text{path}_l}=f_r(\text{path}_l)$5 to $\hat y_{\text{path}_l}=f_r(\text{path}_l)$6, and on DeepSeek-R1-Distill-Qwen-7B the MATH500 score rises from $\hat y_{\text{path}_l}=f_r(\text{path}_l)$7 to $\hat y_{\text{path}_l}=f_r(\text{path}_l)$8 (Yang et al., 17 Feb 2025).

For general instruction following, hidden-thought TCPO improves over direct answer-only preference optimization after iterative training. In "Thinking LLMs," the seed model with a thought prompt initially underperforms the direct baseline, but after preference optimization the TPO model reaches $\hat y_{\text{path}_l}=f_r(\text{path}_l)$9 length-controlled win rate on AlpacaEval 2 and $s_{i-1}=[x,z_1,\dots,z_{i-1}]$0 win rate on Arena-Hard, compared with $s_{i-1}=[x,z_1,\dots,z_{i-1}]$1 and $s_{i-1}=[x,z_1,\dots,z_{i-1}]$2 for a direct response baseline on the same 8B base model (Wu et al., 2024). TUR-DPO shows that adding topology and uncertainty signals improves both reasoning performance and calibration: on GSM8K, the main table reports $s_{i-1}=[x,z_1,\dots,z_{i-1}]$3 EM for TUR-DPO versus $s_{i-1}=[x,z_1,\dots,z_{i-1}]$4 for DPO, and calibration improves from ECE $s_{i-1}=[x,z_1,\dots,z_{i-1}]$5 under DPO to $s_{i-1}=[x,z_1,\dots,z_{i-1}]$6 under TUR-DPO (Abdullah et al., 30 Apr 2026).

In explanation-heavy domain tasks, TCPO-style supervision improves both performance and rationale quality. The science-question scoring framework based on thought trees reports a DPO rationale model with overall QWK $s_{i-1}=[x,z_1,\dots,z_{i-1}]$7, compared with $s_{i-1}=[x,z_1,\dots,z_{i-1}]$8 for AERA and $s_{i-1}=[x,z_1,\dots,z_{i-1}]$9 for Mixtral 8×7B SFT, corresponding to a reported $z_i^{(j)}$0 improvement in QWK relative to AERA; human evaluation also reports key element correctness $z_i^{(j)}$1 versus $z_i^{(j)}$2 for AERA and rubric faithfulness $z_i^{(j)}$3 versus $z_i^{(j)}$4 (Li et al., 2024). RACE-Align, trained on retrieval-grounded TCM CoT trajectories, improves answer quality and thought-style metrics over both a base model and an SFT-only model; in blind human evaluation, it attains $z_i^{(j)}$5 on “TCM thinking pattern application,” $z_i^{(j)}$6 on “Reasoning logicality & depth,” $z_i^{(j)}$7 on “Information richness & insightfulness,” and $z_i^{(j)}$8 on “Interpretability & transparency” (Yan et al., 3 Jun 2025).

Multimodal and embodied experiments indicate that TCPO is not restricted to text-only reasoning. UV-CoT reaches $z_i^{(j)}$9 average score with $z^1,\dots,z^n$0 human bounding-box labels, nearly matching Visual-CoT at $z^1,\dots,z^n$1, and with $z^1,\dots,z^n$2 labels reaches $z^1,\dots,z^n$3, surpassing Visual-CoT; its zero-shot setting on unseen datasets reports an average gain of $z^1,\dots,z^n$4 points over baselines (Zhao et al., 25 Apr 2025). In embodied decision-making, the paper that names the paradigm directly reports an ALFWorld average success rate of $z^1,\dots,z^n$5, compared with $z^1,\dots,z^n$6 for PPO and $z^1,\dots,z^n$7 for D3PO, and a GymCards average success of $z^1,\dots,z^n$8; sample-efficiency analysis shows that reaching $z^1,\dots,z^n$9 success requires $z^k$0 steps for TCPO versus $z^k$1 for PPO (Jiao et al., 10 Sep 2025).

6. Limitations, misconceptions, and open directions

A common misconception is that TCPO necessarily means human-labeled chain-of-thought supervision. The surveyed systems show otherwise: preferences may be fully synthetic and rubric-derived, evaluator-derived, search-derived, retrieval-derived, or environment-derived (Li et al., 2024, Wu et al., 2024, Zhao et al., 25 Apr 2025, Jiao et al., 10 Sep 2025). Another misconception is that TCPO always supervises fine-grained steps. Some methods are explicitly stepwise, such as CPO, PORT, and embodied TCPO, but others operate on whole rationales or entire trajectories, such as ThinkPO, RACE-Align, and hidden-thought TPO (Zhang et al., 2024, Lahlou et al., 2024, Yang et al., 17 Feb 2025, Yan et al., 3 Jun 2025).

The main limitations arise from data construction, structural extraction, and scalability. Thought-tree methods can incur exponential search growth—worst-case $z^k$2 paths—and the science-scoring paper reports GPT-4 cost exceeding \$2k for ASAP 1 alone (Li et al., 2024). RACE-Align is sensitive to retrieval quality and was evaluated in one vertical domain with $z^k$3 preference pairs and only five TCM-trained annotators (Yan et al., 3 Jun 2025). TUR-DPO depends on accurate topology extraction and domain-specific verifiers; the paper notes small graphs, limited re-elicitation $z^k$4, and possible failure when graph extraction misses or merges claims (Abdullah et al., 30 Apr 2026). UV-CoT shows a substantial gap between learned visual chains and the upper bound obtained with ground-truth bounding boxes, which suggests that box selection rather than preference learning may be the dominant remaining bottleneck (Zhao et al., 25 Apr 2025). Embodied TCPO still requires online interaction, assumes Markovian dynamics, and is evaluated only on GymCards and ALFWorld (Jiao et al., 10 Sep 2025).

Several papers also identify failure modes that are specific to thought-centric training. TPO reports catastrophic forgetting on HumanEval for Qwen models and skewed reward distributions in generated preference trees (Liao et al., 2024). "Thinking LLMs" documents overthinking, format violations, and math/coding degradation in a general-instruction setting (Wu et al., 2024). ThinkPO reports hyperparameter sensitivity and shows that excessively large length gaps between chosen and rejected CoTs can hurt performance, indicating that “longer reasoning” is not automatically a better preference target (Yang et al., 17 Feb 2025). PORT notes that negative-step construction is itself a modeling bottleneck, since weak-LLM negatives can accidentally be correct and digit corruption targets only a narrow error class (Lahlou et al., 2024).

Open directions in the literature converge on richer and more controlled thought supervision. These include step-wise thought-level preferences beyond whole-rationale labels, human-in-the-loop calibration of synthetic preferences, more efficient exploration than full DFS over thought trees, integration of actual retrieval at inference time rather than only during data construction, stronger automated reasoning metrics, and extensions from trees to graphs, DAGs, or multimodal structures (Li et al., 2024, Yan et al., 3 Jun 2025, Liao et al., 2024, Abdullah et al., 30 Apr 2026). This suggests that TCPO is best understood not as a single algorithm, but as a general alignment pattern: make task-relevant reasoning explicit, derive preferences over those reasoning structures, and optimize the model so that preferred ways of thinking become more probable than dispreferred ones.