ThinkProbe: Structural Reasoning Analysis

• ThinkProbe is a framework for analyzing open-ended LLM reasoning traces by converting free-form <think> text into structured Thought Graphs with eight node and six edge types.
• It uses rule-based segmentation and discriminative semantic linking to compute a comprehensive 19-metric, five-dimensional cognitive profile covering Breadth, Depth, Structure, Metacognition, and Efficiency.
• Evaluated on 4,200 traces from 7 models across 200 open-ended questions, the framework demonstrates that reasoning structure is a stable model-level property beyond final-answer accuracy.

ThinkProbe is a framework for structural analysis of open-ended large-language-model reasoning traces. It converts free-form > traces into Thought Graphs—directed graphs with cycles, eight node types, and six edge types—and derives a 19-metric five-dimensional cognitive profile spanning Breadth, Depth, Structure, Metacognitive behavior, and Efficiency. The framework is explicitly fully non-generative: it combines rule-based segmentation with discriminative semantic linking rather than using one LLM to interpret another. It was evaluated on 4,200 traces from 7 native reasoning models across 200 open-ended questions in 10 cognitive domains, with the central claim that reasoning structure is a stable model-level property not reducible to final-answer accuracy alone (Kerkouri et al., 27 Jun 2026).

1. Conceptual scope and evaluative motivation

ThinkProbe is designed for settings in which correctness is absent, underdetermined, or not the primary evaluative object. The motivating claim is that accuracy-based evaluation becomes inadequate beyond closed benchmarks with known ground truth. In domains such as ethical dilemmas, strategic planning, philosophy, and creative ideation, there may be no single correct answer; the directly observable object of interest is therefore the reasoning trace itself, especially the extended pre-answer deliberation emitted by native reasoning models inside <think> tags (Kerkouri et al., 27 Jun 2026).

Within that setting, ThinkProbe performs structural profiling rather than answer scoring. Its target is not semantic content alone, nor textual style alone, but the organization of cognitive moves expressed across a trace: branching, elaboration, synthesis, revision, critique, reframing, and convergence. A central implication is that two models may be equally successful at producing acceptable final answers while differing materially in the geometry of the reasoning they express. ThinkProbe treats those differences as measurable model properties rather than anecdotal stylistic variation (Kerkouri et al., 27 Jun 2026).

The framework is also defined by what it avoids. “Non-generative” means that no LLM is used to parse, summarize, relabel, or explain another model’s trace. Instead, the pipeline uses deterministic segmentation rules, embedding-based similarity, fixed thresholds, and priority rules for typing nodes and edges. This design is intended to avoid what the paper identifies as LLM-analyzing-LLM circularity, so that the extracted structure reflects the target model’s expressed reasoning rather than interpretive bias introduced by a second generative model (Kerkouri et al., 27 Jun 2026).

2. Thought Graph formalism

A Thought Graph is defined as

$G=(V,E,\lambda_V,\lambda_E),$

where $V=\{v_0,\ldots,v_n\}$ is an ordered set of Thought Units (TUs), $E\subseteq V\times V$ is a set of directed edges, $\lambda_V:V\to\mathcal N$ assigns one of eight node types, and $\lambda_E:E\to\mathcal E$ assigns one of six edge types. A TU is a contiguous span of text expressing a single cognitive move (Kerkouri et al., 27 Jun 2026).

The graph is intentionally directed and cycle-permitting. Cycles are not treated as noise. The framework argues that trees and DAGs fail to represent two recurrent phenomena in long reasoning traces: cross-branch synthesis and iterative return to earlier lines of thought. For depth computations, however, the elaboration-only subgraph remains acyclic. The formal invariants are: no self-loops; valid node references; the elaboration-only subgraph $G_{\text{elab}}=(V,E_{\text{ELAB}})$ must be acyclic; every node must be reachable from $v_0$; and there must be at least one exploration-family node. Traces violating these conditions after extraction are dropped, which occurs in fewer than 0.2% of cases (Kerkouri et al., 27 Jun 2026).

The node taxonomy is grouped into four cognitive families.

Family	Node types
Exploration	HYP, RFR
Elaboration	JUS, SPC
Evaluation	CRT, CMP, MET
Convergence	SYN

HYP denotes hypothesis generation, RFR reframing, JUS justification, SPC specification, CRT critique, CMP comparison, MET metareflection, and SYN synthesis or conclusion. The distinction between RFR and HYP is conceptually important: a reframing remains semantically anchored to the original problem, whereas a hypothesis is a sharper branch into a new line of thought (Kerkouri et al., 27 Jun 2026).

The edge taxonomy contains six relation types: SEQ, BRCH, ELAB, BACK, SYNT, and CRIT. SEQ encodes default sequential flow, BRCH a new branch or direction, ELAB elaboration, BACK revision or backtracking, SYNT synthesis across branches, and CRIT a directed critique relation. The paper notes a minor inconsistency between the main text and appendix regarding whether critique is always represented by a separate CRIT edge or in some cases by BACK plus a CRT node, but the intended taxonomy is consistently given as six edge types (Kerkouri et al., 27 Jun 2026).

3. Extraction pipeline and discriminative semantic linking

ThinkProbe begins with rule-based segmentation. Markdown headers and bold standalone lines induce branch boundaries; list items each become TUs; blank lines and certain paragraph transitions become separators. Cue phrases in the first sentence of a span assign boundary classes such as meta, convergence, and backtrack. Very short spans are merged by heuristic rules. For long residual spans, the framework applies a TextTiling-style soft segmentation using sentence embeddings from all-MiniLM-L6-v2. A sliding window of width 3 computes left/right block cosine similarities, and local minima with prominence at least 0.15 below a per-trace adaptive threshold $\tau$ become new boundaries, where $\tau$ is the 30th percentile of within-trace inter-sentence cosine similarities (Kerkouri et al., 27 Jun 2026).

After segmentation, adjacent TUs are linked semantically. For each adjacent pair $(v_{i-1},v_i)$, cosine similarity $V=\{v_0,\ldots,v_n\}$0 is computed and two per-trace thresholds are derived:

$V=\{v_0,\ldots,v_n\}$1

If $V=\{v_0,\ldots,v_n\}$2, the edge is labeled BRCH; if $V=\{v_0,\ldots,v_n\}$3, ELAB; otherwise SEQ. Structural and cue-based overrides take precedence: branch-class TUs force BRCH, convergence-class TUs force synthesis, and backtrack-class TUs force backtracking (Kerkouri et al., 27 Jun 2026).

A second layer adds nonlocal edges. For synthesis arcs, if a TU has cosine similarity above 0.50 to the centroids of at least two earlier thematic segments, synthesis links are added to representative TUs of those segments and that TU is promoted to convergence. For revision arcs, each sufficiently late TU $V=\{v_0,\ldots,v_n\}$4 is compared to earlier TUs in different segments; if the most similar earlier TU $V=\{v_0,\ldots,v_n\}$5 satisfies $V=\{v_0,\ldots,v_n\}$6, there is evidence of topic divergence in between, and similarity exceeds 0.65, a back edge $V=\{v_0,\ldots,v_n\}$7 is added. This is the paper’s sense of discriminative semantic linking: edges are inferred by embeddings, thresholds, and deterministic rules rather than by generative interpretation (Kerkouri et al., 27 Jun 2026).

Node typing is likewise deterministic and priority-based. The hierarchy given in the main text is: any TU with outgoing SYNT edges to at least two targets becomes SYN; any TU with outgoing BACK becomes CRT; boundary class meta gives MET; a branch-starting TU with $V=\{v_0,\ldots,v_n\}$8 is RFR; other branch-starting TUs are HYP; contrast boundaries give CMP; support gives JUS; convergence gives SYN; and the default assigns $V=\{v_0,\ldots,v_n\}$9 to HYP and all other nodes to SPC (Kerkouri et al., 27 Jun 2026).

The overall pipeline is therefore: raw <think> text; sentence splitting; hard boundary detection; cue-based boundary typing; short-span merging; embedding-based soft segmentation; adjacent semantic typing; segment partitioning; cross-segment synthesis and revision linking; deterministic node typing; graph assembly; invariant checking; and metric computation. The appendix provides pseudocode consistent with this structure, though the paper notes minor inconsistencies between appendix and main text in some minimum-granularity heuristics (Kerkouri et al., 27 Jun 2026).

4. Five-dimensional cognitive profile

From each Thought Graph, ThinkProbe computes a 5D Cognitive Profile (5D-CP). The five dimensions are Breadth, Depth, Structure, Metacognitive, and Efficiency. Each trace-level profile is built by globally z-scoring each metric over the corpus and averaging within a dimension:

$E\subseteq V\times V$0

Model-level and domain-level profiles are then obtained by averaging trace profiles (Kerkouri et al., 27 Jun 2026).

Breadth captures how widely a model explores the idea space. Its metrics are Branching Factor (BF), Unique Perspectives (UPC), Domain Spread (DS), and First-Idea Diversity (FID). BF is

$E\subseteq V\times V$1

the rate of branch edges per node. UPC counts the number of RFR nodes. DS counts semantic clusters among hypothesis-space embeddings using agglomerative clustering at cosine threshold 0.45; the appendix notation contains V_{BRS}, which the paper itself treats cautiously because elsewhere DS is described as clustering among HYP nodes. FID is the mean pairwise cosine distance among the first three hypothesis nodes (Kerkouri et al., 27 Jun 2026).

Depth measures how far a line of reasoning is developed. It contains Max Elaboration Chain (MEC) and Mean Branch Depth (MBD). MEC is the longest path in the elaboration-only subgraph,

$E\subseteq V\times V$2

while MBD is the mean shortest-path distance from semantic roots in the semantic subgraph $E\subseteq V\times V$3 (Kerkouri et al., 27 Jun 2026).

Structure characterizes branching, convergence, revision, and integration. Its six metrics are Exploration/Exploitation Ratio (EER), Backtracking Rate (BR), Cross-Branch Connectivity (CBC), Convergence Index (CI), Graph Density (GD), and Revision Depth (RvD). Representative definitions are:

$E\subseteq V\times V$4

$E\subseteq V\times V$5

$E\subseteq V\times V$6

CBC measures the fraction of branch pairs connected by at least one synthesis edge. CI measures whether synthesis in-degree is concentrated in synthesis nodes relative to graph-wide average in-degree. RvD is the average positional span $E\subseteq V\times V$7 over backtracking arcs (Kerkouri et al., 27 Jun 2026).

Metacognitive behavior is represented by Critique/Hypothesis Ratio (CHR), Hedging Density (HD), and Perspective Taking (PT). CHR is

$E\subseteq V\times V$8

HD is the fraction of TUs containing uncertainty markers such as “might,” “perhaps,” “possibly,” “likely,” or “unclear,” and PT is

$E\subseteq V\times V$9

Together these metrics are intended to capture self-monitoring, uncertainty marking, critique, and reframing (Kerkouri et al., 27 Jun 2026).

Efficiency contains Token per Idea (TPI), Redundancy Ratio (RR), Avg. Tokens (AT), and Avg. TUs (ATU). TPI is tokens divided by $\lambda_V:V\to\mathcal N$0, with fallback to tokens divided by $\lambda_V:V\to\mathcal N$1 if there are no reframing nodes. RR is the fraction of TU pairs with cosine similarity above 0.75. AT and ATU are mean token count and mean number of thought units per trace. The appendix labels AT and ATU as “Summary,” but in the reported 5D profile they function as Efficiency-family metrics (Kerkouri et al., 27 Jun 2026).

5. Corpus, protocol, and empirical findings

ThinkProbe was applied to 4,200 traces from 7 native reasoning models—GLM-4.7-Flash, Phi-4-reasoning, Gemma-4-31B, Qwen3.5-35B-A3B, Mistral-Medium-3.5, Nemotron-Super-120B-A12B, and GPT-OSS-120B—over 200 open-ended questions in 10 cognitive domains, with 3 runs per model-question pair. The domains are Ethical Dilemmas, Policy Design, Strategic Planning, Scientific Speculation, Creative Problem Solving, Interpersonal Reasoning, Economics/Markets, Geopolitics, Environmental Ethics, and Philosophy/Metaphysics. Questions averaged 87 words and were manually authored with LLM assistance, then reviewed by two expert annotators under a binary criterion requiring genuine open-endedness and the absence of a single correct answer (Kerkouri et al., 27 Jun 2026).

Collection conditions were standardized. All models were queried through a unified OpenAI-compatible endpoint, with local hosting through vLLM where needed. The <think> block was extracted verbatim and the final answer discarded. Hyperparameters were fixed across models: temperature 0.7, max tokens 20,000, empty prompt variant, 3 runs per question, loop detection via repeated normalized-line hashing, up to 3 retries on looping, and truncation at the first repeated segment if looping persisted. A prompt-variant study on Qwen3.5-35B-A3B found that empty, “helpful assistant,” and “think carefully” prompts produced nearly identical structural profiles—for example BF 0.42/0.43/0.43 and UPC 5.6/5.2/4.7—whereas an explicit 7-step elicitation prompt substantially inflated verbosity and UPC (Kerkouri et al., 27 Jun 2026).

All 19 metrics significantly discriminated among models. Kruskal–Wallis effect sizes $\lambda_V:V\to\mathcal N$2 ranged from 0.75 for Avg. Tokens to 0.10 for First-Idea Diversity, with all $\lambda_V:V\to\mathcal N$3 and all surviving Bonferroni correction. Importantly, large effects were not confined to length proxies: Perspective Taking, Critique/Hypothesis Ratio, and Unique Perspectives each reached $\lambda_V:V\to\mathcal N$4, while Backtracking Rate reached 0.43. This supports the claim that model differentiation is not reducible to verbosity alone (Kerkouri et al., 27 Jun 2026).

At the 5D-profile level, model differences were qualitatively distinct. Phi-4-reasoning was highest on Efficiency with $\lambda_V:V\to\mathcal N$5, and also above average on Depth ($\lambda_V:V\to\mathcal N$6), Structure ($\lambda_V:V\to\mathcal N$7), and Metacognitive ($\lambda_V:V\to\mathcal N$8), while below average on Breadth ($\lambda_V:V\to\mathcal N$9); it is described as deep, structured, and economical rather than exploratory. GLM-4.7-Flash led on Breadth ($\lambda_E:E\to\mathcal E$0) and Depth ($\lambda_E:E\to\mathcal E$1), with near-zero Structure and below-average Efficiency and Metacognitive scores. GPT-OSS-120B was also breadth-forward but flatter overall. Gemma-4-31B was lowest on Breadth ($\lambda_E:E\to\mathcal E$2), Depth ($\lambda_E:E\to\mathcal E$3), and Efficiency ($\lambda_E:E\to\mathcal E$4), with the shortest and simplest traces. Qwen3.5-35B-A3B, Mistral-Medium-3.5, and Nemotron-120B occupied more moderate positions near the origin (Kerkouri et al., 27 Jun 2026).

The paper’s strongest interpretive result is that reasoning structure is a stable model-level property. Grouping by domain rather than model, between-model variance exceeded between-domain variance across four of the five dimensions. The reported model/domain effect-size ratios were 6.3× for Breadth, 4.8× for Depth, 2.0× for Metacognitive, and 4.7× for Efficiency in the full profile; Efficiency remained 3.8× after removing length-related metrics. The highest model-level effect among all 19 metrics was $\lambda_E:E\to\mathcal E$5, whereas the highest domain-level effect was only $\lambda_E:E\to\mathcal E$6 for Hedging Density. For Branching Factor and Graph Density, domain effects were near zero ($\lambda_E:E\to\mathcal E$7) while model effects exceeded 0.22. The main exception was Structure, where domain sensitivity was genuine: model $\lambda_E:E\to\mathcal E$8 and domain $\lambda_E:E\to\mathcal E$9 (Kerkouri et al., 27 Jun 2026).

6. Validation, cycle structure, limitations, and significance

Ablation studies indicate that each pipeline layer contributes nonredundant signal. On a held-out set of 50 traces, structural boundaries alone yielded zero for typed-edge metrics by construction. Adding the semantic-trajectory layer raised Branching Factor to 0.22, Max Elaboration Chain to 3.18, and Graph Density to 0.17. Adding the cross-segment layer activated Revision Depth from 0 to 3.87 and initialized Backtracking Rate and Cross-Branch Connectivity. The full pipeline then raised Cross-Branch Connectivity from 0.05 to 0.22 and Graph Density to 0.30. These results support the claim that long-range linking is necessary for the graph structures ThinkProbe is intended to represent (Kerkouri et al., 27 Jun 2026).

The appendix’s cycle analysis is a direct validation of the decision to use directed cyclic graphs. Directed cycles appeared in 95.6% of traces, that is 4,014 / 4,200. The distribution was strongly right-skewed, with median 32 cycles, mean 49.1, and a maximum of 7,051 cycles in the most recursive trace. Across the corpus there were 206,429 cycles with median cycle length 14 nodes, indicating substantial multi-step loops rather than trivial artifacts. Model-level differences aligned with the broader profiles: GLM-4.7-Flash had the highest mean cycle count (75.4, 100% prevalence), while Gemma-4-31B was a low-recursion outlier (9.6, 84.3% prevalence). The paper further reports that cycles arise almost entirely from SYNT edges interacting with the sequential backbone, not from the elaboration-plus-backtrack subgraph, which remains acyclic by design (Kerkouri et al., 27 Jun 2026).

Human validation was limited but informative. In a 50-trace annotation study with two expert annotators, the pipeline was compared to manual segmentation. Because annotators differed substantially in granularity—mean 13.6 TUs per trace for one annotator, 22.4 for the second, and 37.2 for the pipeline—the evaluation emphasized boundary F1 with a $G_{\text{elab}}=(V,E_{\text{ELAB}})$0-sentence tolerance. On that measure, the pipeline achieved 0.705 F1 against the medium-granularity annotator, exceeding inter-annotator F1 of 0.582. At the same tolerance, at least 98% of human-identified major boundaries were recovered by the pipeline (Kerkouri et al., 27 Jun 2026).

The framework’s limitations are explicitly acknowledged. The rule-based node classifier is cue-sensitive, so markers such as “however” or “but” may be over-assigned to backtracking. The pipeline depends on a single embedding model, all-MiniLM-L6-v2, and robustness to alternative encoders has not been empirically established. There is a length confound, especially through Avg. Tokens, Avg. TUs, and Token/Idea, although recomputing the 5D profile without those metrics left four dimensions unchanged and reduced Efficiency by at most 14%, while preserving model-over-domain dominance. Equal-weight aggregation may be simplistic, especially for Structure, whose rankings are sensitive under PCA-weighted alternatives. Human validation remains small-scale. The method also depends on the availability and format of <think> traces, and the paper emphasizes that trace faithfulness is not guaranteed: ThinkProbe measures the structure of expressed reasoning, not necessarily the model’s latent internal mechanism. Finally, the framework does not yet correlate its 5D dimensions with external human judgments such as coherence, novelty, or persuasiveness (Kerkouri et al., 27 Jun 2026).

Within those bounds, ThinkProbe provides an instrumentation layer for reasoning analysis in open-ended settings. The paper identifies potential applications including profile-aware model selection, deployment monitoring, outlier detection for reasoning failures invisible at output level, and possible future real-time structural confidence signals during inference. Its broader significance lies in treating reasoning style as a measurable property of LLMs: not merely whether an answer is acceptable, but how branching, elaboration, revision, synthesis, critique, and reframing are organized across the trace (Kerkouri et al., 27 Jun 2026).