CopT: Contrastive On-Policy Thinking

• CopT is a reasoning pipeline that first generates a draft answer and then employs on-policy thinking for reflection and correction.
• It uses a reverse-KL contrastive verification between discrete tokens and continuous embeddings to assess answer reliability dynamically.
• Empirical results show that CopT improves peak accuracy up to 23% and reduces token usage by up to 57% across varied benchmarks.

CopT is a reformulated reasoning pipeline for LLMs that reverses the usual order of thinking and answering. Instead of thinking before answering, CopT first elicits a draft answer and then invokes subsequent on-policy thinking conditioned on its own draft answer for reflection and correction. To assess whether the draft answer should be trusted, CopT recasts continuous embeddings as inference-time contrastive verifiers, contrasting the model’s support for the same generated tokens under discrete-token inputs and continuous-embedding inputs. Across mathematics, coding, and agentic reasoning tasks, it improves peak accuracy by up to 23% and reduces token usage by up to 57%, without any additional training (Shi et al., 19 May 2026).

1. Motivation and problem setting

Standard Chain-of-Thought (CoT) follows a “think-then-answer” paradigm in which the model generates a full reasoning trace before producing an answer. CopT is motivated by the observation that this ordering can be inefficient when the model is able to identify an answer before extended thinking, a behavior termed performative reasoning. In that regime, the conventional pipeline delays access to a plausible answer and incurs unnecessary token costs even when explicit reasoning is not needed (Shi et al., 19 May 2026).

CopT addresses this by making the draft answer the first object of inference. The draft is not treated as final by default; rather, it becomes the conditioning context for a subsequent reliability test. If the draft appears reliable, it is returned immediately. If it appears insufficiently reliable, the system enters a conditional reflection stage. This design preserves early exit when possible while retaining the capacity for correction on harder instances.

A common misconception is that CopT is merely answer-first prompting. The defining feature is not only the inversion of answer and reasoning order, but also the use of a contrastive reliability criterion and a controlled on-policy refinement stage. The pipeline therefore combines early answer proposal, inference-time verification, and selective deliberation rather than replacing CoT with a single-shot decoder heuristic.

2. Draft-first inference and contrastive verification

In the draft-answer stage, the model is prompted in “answering mode” and generates a draft token sequence $a_1,\ldots,a_{T_a}$. At each step it caches the chosen-token probability and a continuous “soft” embedding,

$$e_t = \sum_{v\in V} p_\theta(v \mid q, a_{<t}) \cdot E(v).$$

The reliability of the draft is then assessed through a sequence-level reverse KL estimator,

$$\kappa_a(a_{1:T_a}) = \frac{1}{T_a}\sum_{t=1}^{T_a} \left[ \log p_\theta(a_t \mid q, a_{<t}) - \log p_\theta(a_t \mid q, e_{<t}) \right].$$

Here, $p_\theta(a_t \mid q, e_{<t})$ is obtained by running the model once with the discrete prefix replaced by the cached soft embeddings. If $\kappa_a \le \tau_a$, the draft is accepted and the procedure stops. Otherwise, CopT proceeds to on-policy thinking (Shi et al., 19 May 2026).

The contrast is between a “student” distribution conditioned on the discrete prefix and a “teacher” distribution conditioned on the continuous prefix. In practice, CopT estimates the reverse KL on the realized trajectory rather than by full enumeration. This estimator is used as an inference-time verifier for answer reliability. The paper’s account emphasizes that the verifier is sequence-level and token-conditioned, rather than a scalar confidence head added by training.

This construction is central to CopT’s computational economy. It allows the model to avoid full reasoning traces on easy examples while still detecting when the draft is unstable under the contrast between discrete-token and continuous-embedding conditioning.

3. On-policy thinking and dynamic visibility control

When the draft is rejected, CopT performs further on-policy thinking. Reasoning is generated in fixed-length chunks of $C$ tokens, and each chunk is associated with a visibility mask $m_k \in \{0,1\}$ indicating whether the draft answer is visible during the next chunk. For chunk $k$, the model generates reasoning tokens conditioned on $(q, a^{(m_k)}, r_{<t})$, again caching token probabilities and continuous embeddings (Shi et al., 19 May 2026).

At the end of each chunk, CopT computes a second KL-based estimator,

$$\kappa_r^{(k)} = \frac{1}{C}\sum_{t=s_k}^{s_k+C-1} \left[ \log p_\theta(r_t \mid q,a^{(m_k)},r_{<t}) - \log p_\theta(r_t \mid q,a^{(m_k)},r_{s_k:t-1}\text{ replaced by }e_{s_k:t-1}) \right].$$

Visibility is then updated according to

$$e_t = \sum_{v\in V} p_\theta(v \mid q, a_{<t}) \cdot E(v).$$0

This means that stable chunks reveal the draft in the next chunk, while unstable chunks hide it. The paper describes this as a mechanism that preserves useful partial information while reducing the risk of being misled by unreliable content. The resulting control loop is therefore not a binary choice between “use the draft” and “discard the draft,” but a dynamic policy over chunked reasoning.

The phrase “on-policy” is substantive here: the reasoning process is conditioned on the model’s own emitted draft answer rather than on an externally supplied candidate. CopT therefore treats the model’s initial output as an endogenous hypothesis to be interrogated, retained, or partially suppressed during subsequent reflection.

4. Theoretical interpretation

CopT’s theoretical analysis studies the reverse-KL estimator under a mixture-linear prefix assumption. At a single step, the estimator can be written as

$$e_t = \sum_{v\in V} p_\theta(v \mid q, a_{<t}) \cdot E(v).$$1

and its expectation satisfies

$$e_t = \sum_{v\in V} p_\theta(v \mid q, a_{<t}) \cdot E(v).$$2

Under these assumptions, the expected estimate equals the mutual information between the unresolved latent state and the emitted answer token (Shi et al., 19 May 2026).

This result explains why the estimator captures answer-relevant uncertainty rather than arbitrary uncertainty in the latent state. The distinction is important because many uncertainty surrogates conflate overall hidden-state variability with the uncertainty specifically coupled to the emitted answer. CopT’s verifier is intended to track the latter.

A plausible implication is that the method is most naturally interpreted as a reliability filter over generated trajectories rather than as a calibrated probability estimator in the conventional sense. The paper’s analysis does not present the estimator as a general confidence score detached from generation; instead, it is tied to the same generated tokens whose trustworthiness is being evaluated.

5. Empirical results and operating characteristics

CopT is evaluated on 10 benchmarks spanning mathematics, coding, and agentic reasoning. The reported gains include both peak-accuracy improvements and matched-accuracy token reductions. On Qwen3-8B, the paper reports gains on GSM8K, Math500, AIME24/25, HumanEval, LeetCode-Contest, and MBPP. On Qwen3.5-35B, it reports a substantial gain on ZebraArena, described as a multi-turn agentic benchmark (Shi et al., 19 May 2026).

Benchmark / setting	Reported result
GSM8K	$$e_t = \sum_{v\in V} p_\theta(v \mid q, a_{<t}) \cdot E(v).$$3 peak accuracy, $$e_t = \sum_{v\in V} p_\theta(v \mid q, a_{<t}) \cdot E(v).$$4 tokens at matched accuracy
Math500	$$e_t = \sum_{v\in V} p_\theta(v \mid q, a_{<t}) \cdot E(v).$$5 peak, $$e_t = \sum_{v\in V} p_\theta(v \mid q, a_{<t}) \cdot E(v).$$6 tokens
HumanEval	$$e_t = \sum_{v\in V} p_\theta(v \mid q, a_{<t}) \cdot E(v).$$7 peak, $$e_t = \sum_{v\in V} p_\theta(v \mid q, a_{<t}) \cdot E(v).$$8 tokens
ZebraArena	$$e_t = \sum_{v\in V} p_\theta(v \mid q, a_{<t}) \cdot E(v).$$9 peak accuracy, $$\kappa_a(a_{1:T_a}) = \frac{1}{T_a}\sum_{t=1}^{T_a} \left[ \log p_\theta(a_t \mid q, a_{<t}) - \log p_\theta(a_t \mid q, e_{<t}) \right].$$0 tokens

The aggregate summary is “up to $$\kappa_a(a_{1:T_a}) = \frac{1}{T_a}\sum_{t=1}^{T_a} \left[ \log p_\theta(a_t \mid q, a_{<t}) - \log p_\theta(a_t \mid q, e_{<t}) \right].$$1 absolute accuracy gain on hard agentic tasks and up to $$\kappa_a(a_{1:T_a}) = \frac{1}{T_a}\sum_{t=1}^{T_a} \left[ \log p_\theta(a_t \mid q, a_{<t}) - \log p_\theta(a_t \mid q, e_{<t}) \right].$$2 token reduction on easier tasks at comparable accuracy.” The paper also reports $$\kappa_a(a_{1:T_a}) = \frac{1}{T_a}\sum_{t=1}^{T_a} \left[ \log p_\theta(a_t \mid q, a_{<t}) - \log p_\theta(a_t \mid q, e_{<t}) \right].$$3–$$\kappa_a(a_{1:T_a}) = \frac{1}{T_a}\sum_{t=1}^{T_a} \left[ \log p_\theta(a_t \mid q, a_{<t}) - \log p_\theta(a_t \mid q, e_{<t}) \right].$$4 end-to-end latency reductions on GPU. These gains are achieved without any additional training, which places CopT in the category of inference-time reasoning methods rather than post-training or finetuning procedures (Shi et al., 19 May 2026).

Ablations reported in the paper further state that the $$\kappa_a(a_{1:T_a}) = \frac{1}{T_a}\sum_{t=1}^{T_a} \left[ \log p_\theta(a_t \mid q, a_{<t}) - \log p_\theta(a_t \mid q, e_{<t}) \right].$$5 estimator pinpoints erroneous drafts far better than uniform sampling of examples for extra thinking, and that the visibility threshold $$\kappa_a(a_{1:T_a}) = \frac{1}{T_a}\sum_{t=1}^{T_a} \left[ \log p_\theta(a_t \mid q, a_{<t}) - \log p_\theta(a_t \mid q, e_{<t}) \right].$$6 trades off correction rate against token usage. The empirical role of the verifier is therefore not merely diagnostic; it directly determines when additional reasoning is invoked and how aggressively the draft remains exposed during that reasoning.

6. Scope, relation to adjacent work, and nomenclature

CopT should be distinguished from similarly named methods in other subfields. “CoPT” denotes “Covariance-based Pixel-Text loss,” a method for unsupervised domain adaptive segmentation using domain-agnostic text embeddings (Mata et al., 8 Jul 2025). “COPT” denotes “Coordinated Optimal Transport,” a graph distance and sketching framework based on a coordinated pair of optimal transport maps (Dong et al., 2020). In contrast, CopT refers specifically to “Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning” (Shi et al., 19 May 2026).

Within the LLM-reasoning literature, CopT occupies a specific position: it is a training-free method that reverses “think then answer” to “answer then think on-policy,” and it uses a contrastive KL estimator in continuous embedding space to judge draft reliability. Its core contribution is therefore architectural at inference time, not a new pretraining objective or a supervised verifier model.

The method’s central claim is not that explicit reasoning is unnecessary, but that explicit reasoning should be invoked conditionally. This suggests a broader view of reasoning as a controllable resource rather than a mandatory preamble to every answer. In CopT, the draft answer, the contrastive verifier, and chunk-wise visibility control form an integrated mechanism for trading off reliability, token cost, and latency under a single inference pipeline (Shi et al., 19 May 2026).