--- title: 'ThinkProbe: Structural Reasoning Analysis' url: https://www.emergentmind.com/topics/thinkprobe type: topic --- # ThinkProbe: Structural Reasoning Analysis ThinkProbe is a framework for structural analysis of open-ended large-language-model reasoning traces. It converts free-form `` traces into **Thought Graphs**—directed graphs with cycles, eight node types, and six edge types—and derives a **19-metric five-dimensional cognitive profile** spanning Breadth, Depth, Structure, Metacognitive behavior, and Efficiency. The framework is explicitly **fully non-generative**: it combines rule-based segmentation with discriminative semantic linking rather than using one LLM to interpret another. It was evaluated on **4,200 traces** from **7 native reasoning models** across **200 open-ended questions** in **10 cognitive domains**, with the central claim that reasoning structure is a stable model-level property not reducible to final-answer accuracy alone [2606.29067]. ## 1. Conceptual scope and evaluative motivation ThinkProbe is designed for settings in which correctness is absent, underdetermined, or not the primary evaluative object. The motivating claim is that accuracy-based evaluation becomes inadequate beyond closed benchmarks with known ground truth. In domains such as ethical dilemmas, strategic planning, philosophy, and creative ideation, there may be no single correct answer; the directly observable object of interest is therefore the reasoning trace itself, especially the extended pre-answer deliberation emitted by native reasoning models inside `` tags [2606.29067]. Within that setting, ThinkProbe performs **structural profiling** rather than answer scoring. Its target is not semantic content alone, nor textual style alone, but the organization of cognitive moves expressed across a trace: branching, elaboration, synthesis, revision, critique, reframing, and convergence. A central implication is that two models may be equally successful at producing acceptable final answers while differing materially in the geometry of the reasoning they express. ThinkProbe treats those differences as measurable model properties rather than anecdotal stylistic variation [2606.29067]. The framework is also defined by what it avoids. “Non-generative” means that no LLM is used to parse, summarize, relabel, or explain another model’s trace. Instead, the pipeline uses deterministic segmentation rules, embedding-based similarity, fixed thresholds, and priority rules for typing nodes and edges. This design is intended to avoid what the paper identifies as **LLM-analyzing-LLM circularity**, so that the extracted structure reflects the target model’s expressed reasoning rather than interpretive bias introduced by a second generative model [2606.29067]. ## 2. Thought Graph formalism A Thought Graph is defined as $$ G=(V,E,\lambda_V,\lambda_E), $$ where $V=\{v_0,\ldots,v_n\}$ is an ordered set of **Thought Units** (TUs), $E\subseteq V\times V$ is a set of directed edges, $\lambda_V:V\to\mathcal N$ assigns one of eight node types, and $\lambda_E:E\to\mathcal E$ assigns one of six edge types. A TU is a contiguous span of text expressing a single cognitive move [2606.29067]. The graph is intentionally **directed and cycle-permitting**. Cycles are not treated as noise. The framework argues that trees and DAGs fail to represent two recurrent phenomena in long reasoning traces: **cross-branch synthesis** and **iterative return to earlier lines of thought**. For depth computations, however, the elaboration-only subgraph remains acyclic. The formal invariants are: no self-loops; valid node references; the elaboration-only subgraph $G_{\text{elab}}=(V,E_{\text{ELAB}})$ must be acyclic; every node must be reachable from $v_0$; and there must be at least one exploration-family node. Traces violating these conditions after extraction are dropped, which occurs in **fewer than 0.2%** of cases [2606.29067]. The node taxonomy is grouped into four cognitive families. | Family | Node types | |---|---| | Exploration | `HYP`, `RFR` | | Elaboration | `JUS`, `SPC` | | Evaluation | `CRT`, `CMP`, `MET` | | Convergence | `SYN` | `HYP` denotes hypothesis generation, `RFR` reframing, `JUS` justification, `SPC` specification, `CRT` critique, `CMP` comparison, `MET` metareflection, and `SYN` synthesis or conclusion. The distinction between `RFR` and `HYP` is conceptually important: a reframing remains semantically anchored to the original problem, whereas a hypothesis is a sharper branch into a new line of thought [2606.29067]. The edge taxonomy contains six relation types: `SEQ`, `BRCH`, `ELAB`, `BACK`, `SYNT`, and `CRIT`. `SEQ` encodes default sequential flow, `BRCH` a new branch or direction, `ELAB` elaboration, `BACK` revision or backtracking, `SYNT` synthesis across branches, and `CRIT` a directed critique relation. The paper notes a minor inconsistency between the main text and appendix regarding whether critique is always represented by a separate `CRIT` edge or in some cases by `BACK` plus a `CRT` node, but the intended taxonomy is consistently given as six edge types [2606.29067]. ## 3. Extraction pipeline and discriminative semantic linking ThinkProbe begins with **rule-based segmentation**. Markdown headers and bold standalone lines induce branch boundaries; list items each become TUs; blank lines and certain paragraph transitions become separators. Cue phrases in the first sentence of a span assign boundary classes such as meta, convergence, and backtrack. Very short spans are merged by heuristic rules. For long residual spans, the framework applies a **TextTiling-style soft segmentation** using sentence embeddings from `all-MiniLM-L6-v2`. A sliding window of width 3 computes left/right block cosine similarities, and local minima with prominence at least **0.15** below a per-trace adaptive threshold $\tau$ become new boundaries, where $\tau$ is the **30th percentile** of within-trace inter-sentence cosine similarities [2606.29067]. After segmentation, adjacent TUs are linked semantically. For each adjacent pair $(v_{i-1},v_i)$, cosine similarity $s_i$ is computed and two per-trace thresholds are derived: $$ \tau_{\text{branch}}=\mathrm{Pct}_{25}(\{s_i\}), \qquad \tau_{\text{elab}}=\mathrm{Pct}_{65}(\{s_i\}). $$ If $s_i<\tau_{\text{branch}}$, the edge is labeled `BRCH`; if $s_i\ge \tau_{\text{elab}}$, `ELAB`; otherwise `SEQ`. Structural and cue-based overrides take precedence: branch-class TUs force `BRCH`, convergence-class TUs force synthesis, and backtrack-class TUs force backtracking [2606.29067]. A second layer adds **nonlocal edges**. For synthesis arcs, if a TU has cosine similarity above **0.50** to the centroids of at least two earlier thematic segments, synthesis links are added to representative TUs of those segments and that TU is promoted to convergence. For revision arcs, each sufficiently late TU $v_j$ is compared to earlier TUs in different segments; if the most similar earlier TU $v_i$ satisfies $j-i\ge 4$, there is evidence of topic divergence in between, and similarity exceeds **0.65**, a back edge $v_j\to v_i$ is added. This is the paper’s sense of **discriminative semantic linking**: edges are inferred by embeddings, thresholds, and deterministic rules rather than by generative interpretation [2606.29067]. Node typing is likewise deterministic and priority-based. The hierarchy given in the main text is: any TU with outgoing `SYNT` edges to at least two targets becomes `SYN`; any TU with outgoing `BACK` becomes `CRT`; boundary class `meta` gives `MET`; a branch-starting TU with $0.25<\cos(\mathbf e_v,\mathbf e_{v_0})<0.72$ is `RFR`; other branch-starting TUs are `HYP`; contrast boundaries give `CMP`; support gives `JUS`; convergence gives `SYN`; and the default assigns $v_0$ to `HYP` and all other nodes to `SPC` [2606.29067]. The overall pipeline is therefore: raw `` text; sentence splitting; hard boundary detection; cue-based boundary typing; short-span merging; embedding-based soft segmentation; adjacent semantic typing; segment partitioning; cross-segment synthesis and revision linking; deterministic node typing; graph assembly; invariant checking; and metric computation. The appendix provides pseudocode consistent with this structure, though the paper notes minor inconsistencies between appendix and main text in some minimum-granularity heuristics [2606.29067]. ## 4. Five-dimensional cognitive profile From each Thought Graph, ThinkProbe computes a **5D Cognitive Profile (5D-CP)**. The five dimensions are **Breadth**, **Depth**, **Structure**, **Metacognitive**, and **Efficiency**. Each trace-level profile is built by globally z-scoring each metric over the corpus and averaging within a dimension: $$ \mathrm{CP}_d=\frac{1}{|\mathcal{M}_d|}\sum_{m\in\mathcal{M}_d} z_m, \qquad d\in\{\text{Breadth, Depth, Structure, Metacog., Efficiency}\}. $$ Model-level and domain-level profiles are then obtained by averaging trace profiles [2606.29067]. Breadth captures how widely a model explores the idea space. Its metrics are **Branching Factor (BF)**, **Unique Perspectives (UPC)**, **Domain Spread (DS)**, and **First-Idea Diversity (FID)**. BF is $$ |E_{\text{BRCH}}|/\max(|V|,1), $$ the rate of branch edges per node. UPC counts the number of `RFR` nodes. DS counts semantic clusters among hypothesis-space embeddings using agglomerative clustering at cosine threshold **0.45**; the appendix notation contains `V_{BRS}`, which the paper itself treats cautiously because elsewhere DS is described as clustering among `HYP` nodes. FID is the mean pairwise cosine distance among the first three hypothesis nodes [2606.29067]. Depth measures how far a line of reasoning is developed. It contains **Max Elaboration Chain (MEC)** and **Mean Branch Depth (MBD)**. MEC is the longest path in the elaboration-only subgraph, $$ \mathrm{longest\mbox{-}path}(V,E_{\text{ELAB}}), $$ while MBD is the mean shortest-path distance from semantic roots in the semantic subgraph $G_{\text{sem}}=(V,E\setminus E_{\text{SEQ}})$ [2606.29067]. Structure characterizes branching, convergence, revision, and integration. Its six metrics are **Exploration/Exploitation Ratio (EER)**, **Backtracking Rate (BR)**, **Cross-Branch Connectivity (CBC)**, **Convergence Index (CI)**, **Graph Density (GD)**, and **Revision Depth (RvD)**. Representative definitions are: $$ \mathrm{EER}=|V_{\text{EXPL}}|/\max(|V_{\text{ELAB}}|,1), $$ $$ \mathrm{BR}=|E_{\text{BACK}}|/\max(|E_{\text{sem}}|,1), $$ $$ \mathrm{GD}=|E_{\text{sem}}|/\bigl(|V|(|V|-1)\bigr). $$ CBC measures the fraction of branch pairs connected by at least one synthesis edge. CI measures whether synthesis in-degree is concentrated in synthesis nodes relative to graph-wide average in-degree. RvD is the average positional span $|\mathrm{pos}(u)-\mathrm{pos}(v)|$ over backtracking arcs [2606.29067]. Metacognitive behavior is represented by **Critique/Hypothesis Ratio (CHR)**, **Hedging Density (HD)**, and **Perspective Taking (PT)**. CHR is $$ |V_{\text{CRT}}|/\max(|V_{\text{HYP}}|,1), $$ HD is the fraction of TUs containing uncertainty markers such as “might,” “perhaps,” “possibly,” “likely,” or “unclear,” and PT is $$ |V_{\text{RFR}}|/\max(|V|,1). $$ Together these metrics are intended to capture self-monitoring, uncertainty marking, critique, and reframing [2606.29067]. Efficiency contains **Token per Idea (TPI)**, **Redundancy Ratio (RR)**, **Avg. Tokens (AT)**, and **Avg. TUs (ATU)**. TPI is tokens divided by $|V_{\text{RFR}}|$, with fallback to tokens divided by $|V|$ if there are no reframing nodes. RR is the fraction of TU pairs with cosine similarity above **0.75**. AT and ATU are mean token count and mean number of thought units per trace. The appendix labels AT and ATU as “Summary,” but in the reported 5D profile they function as Efficiency-family metrics [2606.29067]. ## 5. Corpus, protocol, and empirical findings ThinkProbe was applied to **4,200 traces** from **7 native reasoning models**—**GLM-4.7-Flash**, **Phi-4-reasoning**, **Gemma-4-31B**, **Qwen3.5-35B-A3B**, **Mistral-Medium-3.5**, **Nemotron-Super-120B-A12B**, and **GPT-OSS-120B**—over **200 open-ended questions** in **10 cognitive domains**, with **3 runs per model-question pair**. The domains are Ethical Dilemmas, Policy Design, Strategic Planning, Scientific Speculation, Creative Problem Solving, Interpersonal Reasoning, Economics/Markets, Geopolitics, Environmental Ethics, and Philosophy/Metaphysics. Questions averaged **87 words** and were manually authored with LLM assistance, then reviewed by two expert annotators under a binary criterion requiring genuine open-endedness and the absence of a single correct answer [2606.29067]. Collection conditions were standardized. All models were queried through a unified OpenAI-compatible endpoint, with local hosting through **vLLM** where needed. The `` block was extracted verbatim and the final answer discarded. Hyperparameters were fixed across models: **temperature 0.7**, **max tokens 20,000**, **empty prompt variant**, **3 runs per question**, loop detection via repeated normalized-line hashing, **up to 3 retries** on looping, and truncation at the first repeated segment if looping persisted. A prompt-variant study on **Qwen3.5-35B-A3B** found that empty, “helpful assistant,” and “think carefully” prompts produced nearly identical structural profiles—for example **BF 0.42/0.43/0.43** and **UPC 5.6/5.2/4.7**—whereas an explicit 7-step elicitation prompt substantially inflated verbosity and UPC [2606.29067]. All **19 metrics significantly discriminated among models**. Kruskal–Wallis effect sizes $\varepsilon^2$ ranged from **0.75** for Avg. Tokens to **0.10** for First-Idea Diversity, with all $p<0.001$ and all surviving Bonferroni correction. Importantly, large effects were not confined to length proxies: **Perspective Taking**, **Critique/Hypothesis Ratio**, and **Unique Perspectives** each reached $\varepsilon^2=0.35$, while **Backtracking Rate** reached **0.43**. This supports the claim that model differentiation is not reducible to verbosity alone [2606.29067]. At the 5D-profile level, model differences were qualitatively distinct. **Phi-4-reasoning** was highest on **Efficiency** with $z=+1.27$, and also above average on Depth ($+0.31$), Structure ($+0.19$), and Metacognitive ($+0.29$), while below average on Breadth ($-0.30$); it is described as deep, structured, and economical rather than exploratory. **GLM-4.7-Flash** led on **Breadth** ($+0.66$) and **Depth** ($+0.59$), with near-zero Structure and below-average Efficiency and Metacognitive scores. **GPT-OSS-120B** was also breadth-forward but flatter overall. **Gemma-4-31B** was lowest on Breadth ($-0.38$), Depth ($-0.49$), and Efficiency ($-0.34$), with the shortest and simplest traces. **Qwen3.5-35B-A3B**, **Mistral-Medium-3.5**, and **Nemotron-120B** occupied more moderate positions near the origin [2606.29067]. The paper’s strongest interpretive result is that **reasoning structure is a stable model-level property**. Grouping by domain rather than model, between-model variance exceeded between-domain variance across **four of the five dimensions**. The reported model/domain effect-size ratios were **6.3×** for Breadth, **4.8×** for Depth, **2.0×** for Metacognitive, and **4.7×** for Efficiency in the full profile; Efficiency remained **3.8×** after removing length-related metrics. The highest model-level effect among all 19 metrics was $\varepsilon^2=0.75$, whereas the highest domain-level effect was only $\varepsilon^2=0.19$ for Hedging Density. For **Branching Factor** and **Graph Density**, domain effects were near zero ($<0.02$) while model effects exceeded **0.22**. The main exception was **Structure**, where domain sensitivity was genuine: model $\varepsilon^2=0.121$ and domain $\varepsilon^2=0.156$ [2606.29067]. ## 6. Validation, cycle structure, limitations, and significance Ablation studies indicate that each pipeline layer contributes nonredundant signal. On a held-out set of **50 traces**, structural boundaries alone yielded zero for typed-edge metrics by construction. Adding the semantic-trajectory layer raised **Branching Factor** to **0.22**, **Max Elaboration Chain** to **3.18**, and **Graph Density** to **0.17**. Adding the cross-segment layer activated **Revision Depth** from **0** to **3.87** and initialized **Backtracking Rate** and **Cross-Branch Connectivity**. The full pipeline then raised **Cross-Branch Connectivity** from **0.05** to **0.22** and **Graph Density** to **0.30**. These results support the claim that long-range linking is necessary for the graph structures ThinkProbe is intended to represent [2606.29067]. The appendix’s cycle analysis is a direct validation of the decision to use directed cyclic graphs. Directed cycles appeared in **95.6%** of traces, that is **4,014 / 4,200**. The distribution was strongly right-skewed, with median **32** cycles, mean **49.1**, and a maximum of **7,051** cycles in the most recursive trace. Across the corpus there were **206,429** cycles with median cycle length **14 nodes**, indicating substantial multi-step loops rather than trivial artifacts. Model-level differences aligned with the broader profiles: **GLM-4.7-Flash** had the highest mean cycle count (**75.4**, **100%** prevalence), while **Gemma-4-31B** was a low-recursion outlier (**9.6**, **84.3%** prevalence). The paper further reports that cycles arise almost entirely from `SYNT` edges interacting with the sequential backbone, not from the elaboration-plus-backtrack subgraph, which remains acyclic by design [2606.29067]. Human validation was limited but informative. In a **50-trace** annotation study with **two expert annotators**, the pipeline was compared to manual segmentation. Because annotators differed substantially in granularity—mean **13.6** TUs per trace for one annotator, **22.4** for the second, and **37.2** for the pipeline—the evaluation emphasized boundary F1 with a $\pm 2$-sentence tolerance. On that measure, the pipeline achieved **0.705** F1 against the medium-granularity annotator, exceeding inter-annotator F1 of **0.582**. At the same tolerance, at least **98%** of human-identified major boundaries were recovered by the pipeline [2606.29067]. The framework’s limitations are explicitly acknowledged. The **rule-based node classifier** is cue-sensitive, so markers such as “however” or “but” may be over-assigned to backtracking. The pipeline depends on a **single embedding model**, `all-MiniLM-L6-v2`, and robustness to alternative encoders has not been empirically established. There is a **length confound**, especially through Avg. Tokens, Avg. TUs, and Token/Idea, although recomputing the 5D profile without those metrics left four dimensions unchanged and reduced Efficiency by at most **14%**, while preserving model-over-domain dominance. Equal-weight aggregation may be simplistic, especially for **Structure**, whose rankings are sensitive under PCA-weighted alternatives. Human validation remains small-scale. The method also depends on the availability and format of `` traces, and the paper emphasizes that **trace faithfulness is not guaranteed**: ThinkProbe measures the structure of **expressed reasoning**, not necessarily the model’s latent internal mechanism. Finally, the framework does not yet correlate its 5D dimensions with external human judgments such as coherence, novelty, or persuasiveness [2606.29067]. Within those bounds, ThinkProbe provides an instrumentation layer for reasoning analysis in open-ended settings. The paper identifies potential applications including **profile-aware model selection**, **deployment monitoring**, **outlier detection for reasoning failures invisible at output level**, and possible future **real-time structural confidence signals during inference**. Its broader significance lies in treating reasoning style as a measurable property of language models: not merely whether an answer is acceptable, but how branching, elaboration, revision, synthesis, critique, and reframing are organized across the trace [2606.29067]. Source: https://www.emergentmind.com/topics/thinkprobe