Question Tree Approach for Socratic Debugging

• The Question Tree Approach is a structured framework for Socratic debugging that models a student's progression through a set of necessary knowledge states.
• It uses explicit state representation and dynamic question generation—via sibling and child questions—to address both conceptual and syntactical bugs.
• Empirical results indicate that this tree-based strategy enhances instructional efficiency and dialogue quality in multi-bug debugging scenarios.

The Question Tree Approach is a planning-and-questioning framework for Socratic code debugging in which a debugging dialogue is modeled as traversal through a structured state space rather than as one-shot answer generation. In TreeInstruct, the approach is designed to make a LLM behave like an instructor rather than an assistant: instead of revealing the fix directly, it asks probing questions that adapt to a student’s conceptual and syntactical knowledge, tracks unresolved misunderstandings as explicit state variables, and manages multi-turn interaction across both independent and dependent bugs (Kargupta et al., 2024).

1. Conceptual basis and instructional objective

The approach is motivated by a mismatch between ordinary LLM tutoring behavior and instructional goals. In the debugging setting, direct-answer behavior is pedagogically undesirable because it can give away solutions, ask irrelevant questions, or fail to adapt after a student responds. This is especially problematic when buggy programs contain multiple errors and when some errors are prerequisites for understanding others. A student may need to resolve a conceptual issue before identifying a syntactic one; if questioning targets the wrong issue too early, the interaction becomes inefficient and less educational (Kargupta et al., 2024).

TreeInstruct addresses this by treating the student’s progress as movement through a set of necessary knowledge states. Some of these states are conceptual and some syntactical, but all are meant to be necessary steps toward understanding and correcting the bug set. The conversation is represented as a tree because instruction may need to branch laterally when a student is wrong, descend when a student is partially correct but has not reached the target understanding, and restart when one bug or target state has been resolved. The resulting dialogue policy is therefore neither flat nor purely sequential; it is structured around prerequisite-sensitive instructional control.

A plausible implication is that the method reframes debugging support as guided epistemic progression rather than answer delivery. In that sense, the question tree is not only a dialogue format but also a model of instructional dependency.

2. State-space formulation and tree representation

The formal state space is given as

$S = \{\tau_1, \tau_2, \ldots, \tau_k\},$

where each $\tau_i$ is a task or state variable corresponding to a necessary step toward understanding and correcting the bug or bugs. Each variable is binary-valued, True or False, depending on whether the student has completed it, and all tasks are initialized to False at the start of interaction. Tasks are ordered by priority, with earlier tasks intended to be prerequisites for later ones. The paper’s Fibonacci example includes states such as understanding the Fibonacci definition, recognizing recursive call behavior, and then correcting the recursive call (Kargupta et al., 2024).

State estimation is assigned to a dedicated Verifier prompt. The state representation is defined as “a list of state attributes, where each attribute denotes a specific task that is NECESSARY for the student to successfully understand and implement the given problem.” A task is included only if it directly addresses at least one bug description and is not already addressed in the student’s buggy code. This makes the representation minimal and non-redundant, while still sufficient to reach the correct code. Conceptual tasks are prioritized over syntactic ones when they are prerequisites.

The Question Tree itself is defined as a tree encoding the student’s path to understanding one specific target state variable $\tau_i$. Nodes are questions. Sibling nodes are alternative questions at the same level intended to solidify the same misunderstanding. Parent-child edges indicate progression toward deeper understanding. Each level $l$ contains questions of roughly similar difficulty and depth, and the last level corresponds to that state variable being resolved.

The main components can be organized as follows:

Component	Function	Key condition
Verifier	Estimates state, checks correctness, explains errors	Operates on responses and code state
Sibling question generation	Rephrases or refocuses at same depth	Used when the student is incorrect
Child question generation	Pushes toward deeper understanding	Used when the student is correct but still incomplete

This organization makes the tree both a representation of misunderstanding structure and a control mechanism for question selection.

3. Interaction algorithm and control flow

The algorithm begins by computing the state representation:

$$S = \{\tau_1, \tau_2, \ldots, \tau_k\} \leftarrow \text{GenerateState}(P, B, C),$$

where $P$ is the problem description, $B$ is the buggy code and bug descriptions, and $C$ is the corrected code and bug fixes. It then initializes the tree level $l \leftarrow 0$, the question container $Q \leftarrow \{l:[]\}$, conversation history $\tau_i$0, and student-proposed fixes $\tau_i$1. The first question is generated by

$\tau_i$2

During interaction, the student answers the current question $\tau_i$3, and the Verifier evaluates the response through

$\tau_i$4

where $\tau_i$5 records whether the answer is correct and $\tau_i$6 explains why it is correct or incorrect. The history is updated with $\tau_i$7, and the question is stored at the current level $\tau_i$8. If the answer is incorrect, the system generates a sibling question:

$\tau_i$9

This preserves the same depth and approximate difficulty while changing phrasing or focus in light of the specific misunderstanding.

If the answer is correct, the system performs a second judgment:

$\tau_i$0

This distinguishes local correctness from sufficient understanding. A correct answer does not automatically imply that the target misunderstanding has been resolved. If the target state remains unresolved, the system generates a child question:

$\tau_i$1

and increments the depth. If the target state is resolved, TreeInstruct asks for student-generated fixes through

$\tau_i$2

then resets the tree and moves to the remaining unresolved tasks.

Termination is defined not merely as conversational stopping, but as either all tasks being resolved or all ground-truth fixes being covered by the student’s suggestions. The system also includes a teaching fallback: after a maximum tree width and depth threshold, it appends the correct answer to a prior question and re-asks the most recent question, shifting from pure Socratic questioning to partial teaching. The paper reports that removing this teaching behavior lowers success and logical flow (Kargupta et al., 2024).

4. Knowledge-state estimation, isomorphism, and multi-bug restructuring

Knowledge-state estimation is central to the method because the Verifier is responsible not only for checking correctness but also for deciding whether a response demonstrates sufficient comprehension of a target topic. The prompts formalize this through “isomorphic” reasoning: a student is considered to understand a target if the response either explicitly mentions the target understanding or implicitly exhibits reasoning that is isomorphic to completing the target task. Isomorphism is defined by three conditions: same conclusion or output, same underlying logical structure or pattern, and convertibility through a series of logical transformations (Kargupta et al., 2024).

The same criterion is used when judging whether a student’s proposed bug fix matches a ground-truth fix, and it also appears in the success metric. This gives the framework a semantics of understanding that is broader than surface-form matching. A plausible implication is that TreeInstruct operationalizes comprehension as structural equivalence of reasoning rather than exact verbal reproduction.

The transition rules for the state space further constrain what counts as a necessary task. A task is necessary if the student would never be able to correct the buggy code without completing it. Tasks that are already implicitly satisfied by the buggy code are excluded. When the student’s correct response indicates that the target state has been reached, the system updates the state and may terminate early if the student’s fixes are isomorphic to all ground-truth fixes.

This mechanism is especially important in multi-bug debugging. Once a target state is resolved, TreeInstruct updates all remaining tasks because one conceptual fix may simultaneously resolve dependent bugs. If unresolved tasks remain, it starts a new tree for the next task. This dynamic restructuring is explicitly described as crucial for handling multiple bugs concurrently. Independent bugs should not be forced into one tree, whereas dependent bugs may require staged resolution; the paper’s example is that misunderstanding recursion may need to be corrected before a student can identify a syntactic bug in the recursive call (Kargupta et al., 2024).

5. Empirical setting and reported findings

The empirical study combines an existing single-bug debugging benchmark with a new multi-bug dataset, MULTI-DEBUG. MULTI-DEBUG contains 150 samples built from 50 programming problems, with 1, 2, and 3 injected bugs per problem. Bugs are annotated as conceptual or syntactical, and each bug has a matching description and fix. This dataset is explicitly intended to expose dependency structure that single-bug datasets do not (Kargupta et al., 2024).

The paper reports state-of-the-art performance on both datasets and characterizes TreeInstruct as a more effective instructor than baselines. It also reports a real-world case study with five students of varying skill levels, which further demonstrates the system’s ability to guide debugging efficiently with minimal turns and highly Socratic questioning. The reported ablations attribute the method’s gains primarily to explicit state-space estimation and tree-based planning: removing state information causes the largest drop in relevance and logical flow, while stronger Verifier models produce better logic and success. The state-space and tree-based design is also reported to improve conversational quality, not only end-task success.

These findings support a specific causal interpretation advanced by the paper itself: the explicit modeling of unresolved tasks is the main reason the question tree works well. The tree alone is therefore not presented as a generic prompting device, but as the traversal policy for a state-estimated instructional process.

6. Position within tree-structured reasoning research

TreeInstruct belongs to a broader class of tree-based systems that decompose complex tasks into structured subproblems, traverse those structures adaptively, and use intermediate judgments to control expansion. Related work applies analogous ideas to knowledge-intensive QA, retrieval, explainable reasoning, peer review, and multimodal inference, although with different objects of decomposition and different control semantics.

ProbTree translates a complex question into a query tree and performs probabilistic reasoning from leaf to root, selecting among child-aggregating QA, open-book QA, and closed-book QA through confidence-based scoring (Cao et al., 2023). Tree of Reviews replaces chain-style retrieval with a question-rooted tree whose nodes are retrieved paragraphs and whose branches are controlled by accept, reject, or search decisions (Jiapeng et al., 2024). TreeRare uses a syntax tree of the question to guide bottom-up retrieval and evidence aggregation for knowledge-intensive QA (Zhang et al., 31 May 2025). QDT represents complex KB questions as a Question Decomposition Tree and uses a two-stage generation procedure to preserve the original wording while inserting decomposition structure (Huang et al., 2023). RoHT builds a Hierarchical Question Decomposition Tree and reasons probabilistically over heterogeneous knowledge sources, including structured KBs and unstructured corpora (Zhang et al., 2023). TreeReview treats scientific peer review as a hierarchical question-answering problem with recursive decomposition and dynamic follow-up expansion (Chang et al., 9 Jun 2025).

This suggests a general research pattern: question trees can serve as retrieval scaffolds, reasoning traces, control structures, or pedagogical interfaces depending on the task. TreeInstruct is distinctive within that pattern because its tree is explicitly tied to student knowledge-state estimation, sibling-versus-child question selection, and the instructional transition from probing to partial teaching. Its contribution is therefore not merely the use of a tree, but the coupling of tree traversal with a model of what the learner must understand next (Kargupta et al., 2024).