Trace-Based Reasoning Task

• Trace-based reasoning is a method that treats intermediate computational steps as first-class entities rather than mere by-products.
• It employs diverse representations—such as sequential, graph-structured, and executable traces—to capture multi-step decision processes.
• Recent approaches emphasize process-level supervision using metrics like efficiency, hallucination, and adaptive recovery, shifting focus from final output to reasoning quality.

Trace-based reasoning refers to methods that treat an intermediate reasoning trace, reasoning trajectory, or execution trace as a primary computational object rather than a disposable by-product of answer generation. Across recent work, the trace may be a sequence of tool actions, a chain-of-thought segmented into subthoughts, a directed acyclic graph over reasoning steps, a temporal deduction trace, an executable tool program, or a replayable robot trajectory. The unifying premise is that final-answer correctness alone is insufficient for characterizing reasoning quality, because it obscures efficiency, hallucination, adaptivity, commitment dynamics, faithfulness, reward hacking, and downstream utility (Kim et al., 3 Oct 2025, Hammoud et al., 29 Apr 2025, Ballon et al., 30 Jan 2026).

1. Conceptual scope and defining assumptions

In tool-augmented agents, a reasoning trajectory is defined as the sequence of decisions, thoughts, and tool actions an agent takes to solve a task, not just the final answer. On this view, trace-based reasoning tasks evaluate how the result is reached, including efficiency, hallucination, and adaptivity, rather than only whether the final answer matches an accepted solution (Kim et al., 3 Oct 2025).

Several parallel lines of work generalize this premise. One line studies alternative continuations from intermediate trace segments, arguing that the final answer may not reliably represent the model’s best possible conclusion and that different restart points inside the same trace can yield different answers (Hammoud et al., 29 Apr 2025). Another line probes how answer distributions and decision commitment evolve as progressively larger prefixes of a reasoning trace are supplied back to a model, showing that reasoning traces can be treated as functional interventions rather than merely explanatory text (Ballon et al., 30 Jan 2026). A further line treats traces as long-horizon behavioral objects whose suspiciousness or maliciousness may only emerge through cross-step evidence aggregation rather than per-step inspection or one-shot whole-trajectory scoring (Mittapalli et al., 5 Jun 2026).

This suggests that “trace-based reasoning task” is best understood as an umbrella category covering evaluation, monitoring, optimization, and deployment settings in which intermediate reasoning states are first-class supervisory or diagnostic signals. A plausible implication is that the field is moving from answer-centric benchmarking toward process-centric benchmarking, especially where multi-step tool use, long-horizon planning, or compositional inference makes answer matching too coarse to support diagnosis (Kim et al., 3 Oct 2025, Zhang et al., 9 Feb 2026).

2. Representational forms of reasoning traces

Recent work operationalizes traces in several distinct but related forms.

A sequential formulation treats the trace as an ordered token or step sequence. In subthought-based analysis, a full trace is decomposed by linguistic transition markers into sequential subthoughts,

$T = s_1 \oplus s_2 \oplus \dots \oplus s_n$

and each cumulative prefix is reused to generate a continuation and extract an answer (Hammoud et al., 29 Apr 2025). In trajectory probing, the same basic object is sliced by token-percentiles, producing prefixes such as $$r^{(10)}_{m,i}, r^{(20)}_{m,i}, \dots, r^{(100)}_{m,i}$$, which are re-injected to measure next-token answer probabilities, confidence, non-choice probability, and flip rate (Ballon et al., 30 Jan 2026).

A graph-structured formulation models complex reasoning traces as directed acyclic graphs. In this setting, each atomic reasoning step is a node, edges encode dependencies, and consecutive chains may be collapsed into super-nodes for macro-level abstraction. The resulting representation is used for pairwise evaluation under the ME$^2$ principle, which separates macro- versus micro-level quality and efficiency versus effectiveness (Zhang et al., 9 Feb 2026).

Other domains require executable or stateful traces. In failure management for multi-agent systems, reasoning traces encode both intra-agent reasoning and inter-agent coordination, and are embedded into a latent space via reasoning-scoped contrastive learning to support step-wise anomaly detection and diagnosis (Zhang et al., 23 Mar 2026). In ecological question answering, a full execution trace is explicitly formalized as

$$\mathcal{T} = \big(\hat{a}, \{\mathcal{S}_t\}_{t=0}^{T}, \{a_t\}_{t=0}^{T-1}\big),$$

where $a_t = (o_t, \mathbf{u}_t)$ denotes a deterministic tool invocation and $$(\mathcal{S}_{t+1}, r_t) = f_{o_t}(\mathcal{S}_t, \mathbf{u}_t)$$ denotes the state transition (Cheng et al., 26 May 2026). In temporal reasoning, a trace may instead be a sequence of deduced interval relations whose validity is checked against the closure of a constraint graph derived from Allen’s Interval Algebra (Zhou et al., 6 Jul 2026).

These representations differ in surface form but share a common design choice: they preserve intermediate structure that would be collapsed by outcome-only evaluation. This suggests that trace-based reasoning is not tied to any single trace syntax; what matters is whether intermediate computational commitments remain inspectable, comparable, and reusable (Zhang et al., 9 Feb 2026, Zhou et al., 6 Jul 2026).

3. Evaluation dimensions beyond final-answer correctness

The clearest formulation of multi-dimensional trace evaluation appears in TRACE for tool-augmented agents, which scores final correctness, efficiency, hallucination, and adaptivity. Its efficiency and hallucination metrics are defined over per-step labels:

$$\text{Efficiency}(T) = \frac{|\{s_i \in T: l_E(s_i) = \text{"efficient"}\}|}{|T|}$$

and

$$\text{Hallucination Rate}(T) = \frac{|\{s_i \in T: l_H(s_i) = \text{"hallucination"}\}|}{|T|}.$$

Adaptivity is measured as binary recovery success after injected tool failures (Kim et al., 3 Oct 2025).

Other papers broaden the metric space. In trajectory probing, accuracy and decision commitment consistently increase as larger percentages of reasoning tokens are provided, while flip rate decreases with reasoning depth; the protocol also tracks non-choice probability and distinguishes “lost” trajectories in which longer reasoning replaces a correct answer with a wrong one (Ballon et al., 30 Jan 2026). In trace aggregation from subthoughts, the central comparison is between last-answer accuracy and most-frequent-answer accuracy, with the mode of subthought-derived answers improving performance by up to $13.33\%$ on AIME2024 and up to $10\%$ on AIME2025, while the entropy of the answer distribution is analyzed as a reliability signal (Hammoud et al., 29 Apr 2025).

For open-ended chain-of-thought evaluation without ground truth, TRACE based on Toulmin’s argumentation theory and Flavell’s metacognitive framework scores sentence-level “state validity” and “transition coherence,” then combines them as

$$r^{(10)}_{m,i}, r^{(20)}_{m,i}, \dots, r^{(100)}_{m,i}$$0

with $$r^{(10)}_{m,i}, r^{(20)}_{m,i}, \dots, r^{(100)}_{m,i}$$1 (Kim et al., 28 May 2026). On 26.3K QA samples across 7 reasoning models, the reported correlation with benchmark accuracy is $$r^{(10)}_{m,i}, r^{(20)}_{m,i}, \dots, r^{(100)}_{m,i}$$2 (Kim et al., 28 May 2026).

Monitoring work introduces still another evaluation target: whether long-horizon trajectories exhibit hidden malicious intent. Here the core outcome is a trajectory-level maliciousness verdict after adaptive triage and inspection, with a suspicion score from 1–5 and scores $$r^{(10)}_{m,i}, r^{(20)}_{m,i}, \dots, r^{(100)}_{m,i}$$3 considered malicious. On ten task domains from SHADE-Arena, TRACE achieves aggregate $$r^{(10)}_{m,i}, r^{(20)}_{m,i}, \dots, r^{(100)}_{m,i}$$4 and recall $$r^{(10)}_{m,i}, r^{(20)}_{m,i}, \dots, r^{(100)}_{m,i}$$5, outperforming a Full-Trajectory Monitor and a Sequential Monitor (Mittapalli et al., 5 Jun 2026).

Taken together, these formulations separate at least four families of trace quality: procedural quality, behavioral robustness, epistemic reliability, and oversight utility. A plausible implication is that no single trace metric is likely to dominate across domains, because different tasks expose different failure surfaces: wrong-for-the-right-reason, right-for-the-wrong-reason, silent hallucination, maladaptive recovery, or temporally distributed sabotage (Kim et al., 3 Oct 2025, Han et al., 5 May 2026).

4. Benchmark construction and supervision regimes

Trace-based reasoning research depends heavily on synthetic augmentation, executable environments, or pairwise preference data because raw benchmarks usually provide only a single canonical solution path.

For tool-augmented agent evaluation, a multi-trajectory evidence bank is constructed by augmenting a single canonical ground-truth trajectory into multiple valid solution paths and synthetic flawed trajectories. The resulting meta-evaluation datasets include Meta-GTA with 761 trajectories—168 correct, 171 inefficient, 251 hallucinated, and 171 adaptivity-injected—and Meta-m{paper_content}m’s with 735 trajectories—374 correct and 361 inefficient (Kim et al., 3 Oct 2025). Validation uses Claude Sonnet 4.0, GPT-4o, and Gemini Pro, with consensus filtering retaining only data points agreed upon as valid by all models (Kim et al., 3 Oct 2025).

For complex reasoning optimization, the TRM-Preference dataset is constructed from 64,000 prompts from WebInstruct-verified, retaining only traces whose final answers are verified as correct and then generating pairwise judgments through DAG-guided ME$$r^{(10)}_{m,i}, r^{(20)}_{m,i}, \dots, r^{(100)}_{m,i}$$6-based evaluation. The reported size is 103K training pairs and 1.5K validation pairs (Zhang et al., 9 Feb 2026). These preferences train a Thinking Reward Model with Bradley–Terry loss,

$$r^{(10)}_{m,i}, r^{(20)}_{m,i}, \dots, r^{(100)}_{m,i}$$7

At test time, selecting better reasoning yields up to $$r^{(10)}_{m,i}, r^{(20)}_{m,i}, \dots, r^{(100)}_{m,i}$$8 gain, and during RL training, thinking rewards yield up to $$r^{(10)}_{m,i}, r^{(20)}_{m,i}, \dots, r^{(100)}_{m,i}$$9 gain (Zhang et al., 9 Feb 2026).

Executor-grounded trace supervision sharpens the same theme. TraceLift constructs TRACELIFT-GROUPS from 3,000 OpenCodeReasoning problems and 3,000 GSM8K problems, with each same-problem group containing one high-quality reference trace and multiple plausible flawed traces. Planner reward is defined as

$^2$0

where executor uplift is measured relative to a no-reasoning baseline (Han et al., 5 May 2026).

Dynamic benchmark generation is particularly explicit in temporal reasoning. TRACEBench contains 1,200 synthesized test instances across six target difficulty levels $^2$1, with node counts from 3 to 16 and edge counts from 2 to approximately 60. Pearson’s $^2$2 between model performance and the difficulty metric is reported as evidence that the difficulty control mechanism is effective (Zhou et al., 6 Jul 2026).

These regimes show that trace-based reasoning tasks often require synthetic corruption, alternative path generation, or executable verification. This suggests that trace supervision is as much a dataset construction problem as a model evaluation problem, because valid reasoning is usually non-unique and directly annotating all acceptable traces is prohibitively expensive (Kim et al., 3 Oct 2025, Zhou et al., 6 Jul 2026).

5. Applications across modalities and system types

Trace-based reasoning has rapidly spread beyond text-only question answering.

In retrieval-augmented generation, TRACE constructs knowledge-grounded reasoning chains from KG triples extracted from retrieved documents. The reasoning chain probability is factorized autoregressively as

$^2$3

and experiments on HotPotQA, 2WikiMultiHopQA, and MuSiQue report average EM improvement up to $^2$4 for TRACE-Triple and $^2$5 for TRACE-Doc over a vanilla RAG baseline using all retrieved documents (Fang et al., 2024).

In multimodal retrieval, TRACE generates a structured chain-of-thought and compresses it into an embedding via a dedicated token $^2$6. The model learns an implicit routing behavior that activates reasoning for complex queries and bypasses it for simple ones; during training, M-BEIR-CoT contributes approximately 575K reasoning samples and approximately 518K simple samples (Hao et al., 3 Mar 2026).

In vision-language reasoning, TRACE introduces Auxiliary Reasoning Sets,

$^2$7

and defines path consistency metrics such as Path Mean Consistency, Path Z-score Consistency, and Global Mean Consistency to localize the First Failure Step and define confidence regions separating reliable-correct, reliable-incorrect, and uncertain paths (Imani et al., 5 Dec 2025). In spatially grounded robotics, RoboTracer uses supervised fine-tuning and reinforcement fine-tuning with metric-sensitive process rewards on TraceSpatial, a dataset of 30M QA pairs, and reports an average success rate of $^2$8 as well as performance exceeding Gemini-2.5-Pro by $^2$9 accuracy on TraceSpatial-Bench (Zhou et al., 15 Dec 2025).

In embodied ecological reasoning, ForestHG-Trace represents scenes as ecological hypergraphs and produces replayable execution traces with deterministic tool calls. On ForestTraceQA, ForestHG-Trace reports overall accuracy $$\mathcal{T} = \big(\hat{a}, \{\mathcal{S}_t\}_{t=0}^{T}, \{a_t\}_{t=0}^{T-1}\big),$$0, compared with $$\mathcal{T} = \big(\hat{a}, \{\mathcal{S}_t\}_{t=0}^{T}, \{a_t\}_{t=0}^{T-1}\big),$$1 for a Scene-Graph Agent and $$\mathcal{T} = \big(\hat{a}, \{\mathcal{S}_t\}_{t=0}^{T}, \{a_t\}_{t=0}^{T-1}\big),$$2 for the best single-step baseline, while identifying execution depth as the main bottleneck for long-horizon ecological QA (Cheng et al., 26 May 2026).

In education, tracing is used as a human reasoning scaffold rather than a model-internal supervision signal. In a between-subjects study with 20 students solving the “Jump Game” problem, trace-based planning shifted learners away from code-like, line-by-line descriptions toward more goal-driven reasoning and yielded mean functional correctness $$\mathcal{T} = \big(\hat{a}, \{\mathcal{S}_t\}_{t=0}^{T}, \{a_t\}_{t=0}^{T-1}\big),$$3 versus $$\mathcal{T} = \big(\hat{a}, \{\mathcal{S}_t\}_{t=0}^{T}, \{a_t\}_{t=0}^{T-1}\big),$$4 on a 0–4 scale, although final coding performance remained comparable and LLM feedback quality was not significantly improved (Jain et al., 3 Feb 2026).

These applications indicate that trace-based reasoning is becoming a general systems pattern rather than a narrow evaluation protocol. A plausible implication is that the common substrate is not chain-of-thought visibility per se, but the availability of structured intermediate artifacts that can be verified, compressed, re-ranked, distilled, or executed downstream (Fang et al., 2024, Cheng et al., 26 May 2026).

6. Controversies, failure modes, and limits of trace exposure

A central controversy concerns whether exposed traces are faithful explanations, merely useful interfaces, or exploitable supervision channels.

Human-subject evidence complicates the assumption that showing traces improves reasoning quality for end users. In a preregistered study with $$\mathcal{T} = \big(\hat{a}, \{\mathcal{S}_t\}_{t=0}^{T}, \{a_t\}_{t=0}^{T-1}\big),$$5 on LSAT-style tasks, full-trace exposure impaired objective performance relative to answer-only and summary-trace conditions, while summaries preserved baseline performance and increased trust and hedonic appeal. Across all conditions, participants overestimated their performance, and no trace format improved metacognitive calibration (Fernandes et al., 25 May 2026). This suggests that visible traces can function as persuasive interface artifacts rather than reliable transparency mechanisms.

Security-oriented work raises a different challenge: hidden traces may not remain hidden. Reasoning Exposure Prompting uses shadow-model-generated demonstrations in code-like wrappers to elicit user-visible reasoning traces from victim models. Across several victim models and datasets, REP increases similarity between exposed and REP-conditioned internal traces while preserving useful reasoning signals for distillation; a markdown-fence wrapper with $$\mathcal{T} = \big(\hat{a}, \{\mathcal{S}_t\}_{t=0}^{T}, \{a_t\}_{t=0}^{T-1}\big),$$6 demonstrations gives the strongest reported exposure fidelity in the cited ablation (Lu et al., 30 May 2026). Interface-level suppression of raw traces therefore does not necessarily block reasoning extraction.

Another failure mode is implicit reward hacking. Truncated Reasoning AUC Evaluation measures how early a verifier-passing answer can be produced from a partially truncated chain-of-thought. The TRACE score is the area under the passing-rate-versus-length curve,

$$\mathcal{T} = \big(\hat{a}, \{\mathcal{S}_t\}_{t=0}^{T}, \{a_t\}_{t=0}^{T-1}\big),$$7

On the reported benchmarks, this method achieves over $$\mathcal{T} = \big(\hat{a}, \{\mathcal{S}_t\}_{t=0}^{T}, \{a_t\}_{t=0}^{T-1}\big),$$8 gains over the strongest 72B CoT monitor in math reasoning and over $$\mathcal{T} = \big(\hat{a}, \{\mathcal{S}_t\}_{t=0}^{T}, \{a_t\}_{t=0}^{T-1}\big),$$9 gains over a 32B monitor in coding (Wang et al., 1 Oct 2025). The core claim is that hacking models use less effort than intended-task solving requires, so early truncations already suffice.

Trace-centric evaluation also exposes mismatches between valid reasoning and correct answers. On TRACEBench, trace-based verification reveals a spurious guessing rate of approximately $a_t = (o_t, \mathbf{u}_t)$0 in mid-sized models and identifies scale-dependent failure modes ranging from Degenerative Loops in small models to Reasoning Explosion in advanced architectures (Zhou et al., 6 Jul 2026). More generally, multiple papers explicitly note that a trace can be high-quality without being outcome-correct, or outcome-correct without being high-quality (Zhang et al., 9 Feb 2026, Kim et al., 28 May 2026).

The resulting picture is not that traces are intrinsically trustworthy, but that they are diagnostically valuable when paired with suitable verification, aggregation, or downstream use criteria. This suggests that future work will likely focus less on exposing raw traces and more on defining which trace properties are actionable for evaluation, oversight, and control (Fernandes et al., 25 May 2026, Han et al., 5 May 2026).

7. Research trajectory and emerging synthesis

Current work converges on several recurring principles.

First, process-level supervision is increasingly treated as complementary to outcome-level supervision, not a replacement for it. Tool-agent TRACE couples final correctness with efficiency, hallucination, and adaptivity (Kim et al., 3 Oct 2025); TRM-based optimization combines verifiable rewards with thinking rewards (Zhang et al., 9 Feb 2026); TraceLift explicitly multiplies reasoning quality by executor uplift to ensure that a trace is not only plausible but useful to the module that consumes it (Han et al., 5 May 2026).

Second, cross-step aggregation is becoming central. Evidence banks in tool trajectories, TIJ monitoring for sabotage detection, DAG abstractions for complex reasoning, and experiential priors in multi-hop KGQA all formalize the idea that locally weak signals may become significant only when accumulated across time or structure (Kim et al., 3 Oct 2025, Mittapalli et al., 5 Jun 2026, Wang et al., 13 Apr 2026).

Third, executable and deterministic trace semantics are increasingly favored in domains where free-form text is too ambiguous. This is visible in temporal CSP verification, ecological tool traces, and robotics traces with metric-sensitive process rewards (Zhou et al., 6 Jul 2026, Cheng et al., 26 May 2026, Zhou et al., 15 Dec 2025).

Finally, trace-based reasoning is producing a sharper distinction between reasoning quality, reasoning faithfulness, and reasoning usefulness. A trace may be structurally sound but factually wrong, factually right but spurious, rhetorically compelling but harmful for users, or beneficial only when consumed by a specific downstream executor. A plausible implication is that future benchmark design will treat these as separate targets rather than collapsing them into a single scalar notion of “good reasoning” (Kim et al., 28 May 2026, Fernandes et al., 25 May 2026, Han et al., 5 May 2026).

In that sense, trace-based reasoning tasks mark a methodological shift: they replace the assumption that the final answer is the definitive object of study with the stronger claim that intermediate computational structure is itself measurable, optimizable, and, in many settings, indispensable for understanding what a reasoning system is actually doing (Kim et al., 3 Oct 2025, Hammoud et al., 29 Apr 2025).