Structured Thoughts For Improved Reasoning And Context Pruning

Abstract: LLMs excel at generating long chains of thought, but long reasoning traces are often verbose and memory-inefficient. In this work, we introduce Structured Thoughts, a framework that organizes reasoning into alternating <try> and <outcome> blocks: <try> captures exploratory scratch work, while <outcome> contains the distilled conclusion of that step. We construct a dataset of structured thoughts by segmenting reasoning traces into <try> blocks and prompting an LLM to summarize each step into its corresponding <outcome>. Fine-tuning pretrained foundation models on this reformatted data produces models that adopt the structured reasoning style, leading to performance gains of up to 8.08\% on reasoning benchmarks compared to standard SFT. The explicit structure also enables context pruning: after each <try>/<outcome> pair, the <try> can be pruned, allowing the model to retain conclusions without keeping the full scratch work in the context. A proof-of-concept pruning implementation achieves an average of 85\% memory / context savings with an 8.67\% performance drop across mathematical tasks.

• The paper presents a novel protocol that alternates between <try> and <outcome> blocks, significantly improving reasoning accuracy on mathematical benchmarks.
• The methodology enables dynamic context pruning, achieving up to 85% memory savings while maintaining competitive performance.
• Empirical results on multiple benchmarks validate the structured reasoning approach, underscoring its practical and theoretical impact on LLM efficiency.

Structured Thoughts for Improved LLM Reasoning and Context Pruning

Introduction

This work introduces a novel structured reasoning framework for LLMs by segmenting reasoning traces into alternating <try> and <outcome> blocks. The structure aims to explicitly separate exploratory scratch work from distilled logical conclusions, facilitating both reasoning accuracy and aggressive context pruning. The paper provides a methodology for constructing large-scale structured reasoning datasets, presents fine-tuning and context pruning strategies, and delivers strong empirical results on mathematical benchmarks. The study demonstrates that imposing reasoning structure leads to tangible robustness and accuracy improvements, and that structure-aware pruning affords substantial context/memory savings with moderate accuracy-tradeoff.

Structured Reasoning Formalism

The core contribution is a reasoning protocol in which every problem-solving step is decomposed into a <try> block (containing exploratory, often verbose, scratch computations) followed by a <outcome> block (a concise, summary conclusion for that step). Traces therefore follow this explicit alternation, in contrast to conventionally flat chains-of-thought with minimal demarcation between different phases of reasoning. During annotation, breakpoint heuristics are applied for automatic segmentation, and off-the-shelf instruction models synthesize textual outcome summaries.

Formally, this structure creates natural semantic boundaries within the reasoning trace, offering clear opportunities for identifying and removing unneeded history during generation. The alternation is not merely cosmetic—it enforces information bottlenecks and encourages the model to internalize the habit of extracting and relying on distillations rather than revisiting low-level scratch work.

Implementation of Pruning Procedure

The structured format enables context pruning: after each <try>/<outcome> pair, the <try> tokens are pruned from the autoregressive context. The generation process consists of alternating between step-local computation and storage of high-level conclusions. Once an <outcome> is generated, previous exploratory tokens are deleted from the context passed forward for additional decoding steps.

Figure 1: The structured reasoning/pruning procedure alternates <try> (scratch work, blue) and <outcome> (distilled summary, green) blocks, pruning obsolete scratch work to retain only crucial logical outcomes stepwise.

To support this, the authors implement a custom dynamic masking operator. Training includes a secondary round of SFT where model attention is pruned in an exact match of the inference procedure, so that gradients and attention patterns only reinforce cross-block information routed through <outcome> tokens. For generation, a custom HuggingFace-based autoregressive loop enforces block-level eviction synchronously with completion of each <outcome>.

Empirical Evaluation

Experiments are conducted on two notable reasoning-centric LLMs, Llama-Nemotron-8B and Qwen2.5-7B-Instruct, using the 1M-example Llama-Nemotron math v1.1 corpus with automatic re-structuring. Models are evaluated on a broad math benchmark suite including MATH-500, Minerva-Math, AMC23, AIME24, TheoremQA, OlympiadBench, and GSM8K.

Structured SFT (i.e., supervised fine-tuning on the segmented traces) achieves consistent and significant gains. On Llama-Nemotron-8B, relative improvement is 3.66%; on Qwen2.5-7B-Instruct, improvement is 8.08%. These gains are robust across almost all benchmarks.

Ablations confirm the structure is responsible—simply lengthening reasoning traces without injecting structured summaries does not account for the effect. Compute-matched models (which replace <outcome> content with mask tokens but retain the increased sequence length) deliver a weaker improvement: 1.69%, showing that the separation and summary, rather than added tokens or compute, is the critical factor.

Inference-time pruning, which deletes all <try> tokens after their paired <outcome>, produces mean context/memory savings of 85% while incurring an 8.76% average performance drop relative to unpruned structured SFT. The accuracy tradeoff is attributed to both imperfect segmentation boundaries (since segmentation is heuristic) and incomplete or noisy outcome summarization. Despite this, overall performance remains competitive and supports the hypothesis that most salient logical dependencies are effectively communicated via the outcomes alone.

Practical and Theoretical Implications

This work demonstrates that explicit structure in reasoning traces is a substantial axis of supervision for autoregressive models. Even without advanced reward shaping or RL (as in the recent trend of process-based supervision), hierarchical bottlenecking via reasoning blocks yields improvement. The context pruning protocol exposes a scalable path for long-context reasoning: only high-salience outcomes are propagated, ensuring efficient attention and vastly reduced memory/latency footprint, especially on tasks where scratch computation is dominant.

The study also highlights hardware and runtime barriers. Well-engineered systems for efficient cache eviction and flexible masking (beyond fixed-causal masks as in FlashAttention/vLLM) are not yet mature, constraining the practical deployment of such pruning systems at scale. There is also a mismatch between segmentation heuristics and true logical boundaries, indicating a future direction in learned segmentation or RL-based step detection.

The approach extends beyond mathematics: any domain demanding deep, interpretable, long-horizon reasoning stands to benefit from enforced alternation between computation and summary, with flexible context compression. Theoretical implications include a formalization of information flow in LLMs where intermediate representations become explicit artifacts, rather than implicit or entangled in single long chains.

Prospects for Future AI Research

Future directions involve joint RL over both structure and summary, enabling outcome blocks to be learned concurrently with step boundaries, potentially via meta-optimization or adaptive compute methods. This would minimize reliance on external summarizers and further reduce the performance gap under aggressive pruning. Additionally, system-level improvements enabling blockwise memory management and dynamic pruning will be necessary to render structure-based context compression practical at deployment scales.

The line of research aligns with the movement toward explainable reasoning in LLMs: with explicit, prunable traces, both upstream training dynamics and downstream interpretability benefit. Given that trace length and structure directly impact computational cost and scaling behavior, this methodology may be crucial for next-generation models seeking efficiency at extreme prompt lengths.

Conclusion

The structured thoughts protocol enhances LLM reasoning by framing autoregressive processes as an alternation of computation and outcome, with immediate implications for both performance and context/memory usage. Models trained on such structured traces exhibit measurable accuracy gains on challenging reasoning tasks, outperforming unstructured counterparts. Structure-aware pruning yields major resource efficiency benefits, highlighting the viability of distillation-centric approaches to long-context reasoning. Despite current engineering bottlenecks, the core claim—reasoning structure is an independent, scalable route to stronger, more efficient LLMs—is strongly supported by empirical results and offers new directions for both practical model training and theoretical investigations of LLM computation (2607.10386).