BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning (2501.18858v2)

Abstract: LLMs have demonstrated remarkable capabilities in complex reasoning tasks, yet generating reliable reasoning processes remains a significant challenge. We present a unified probabilistic framework that formalizes LLM reasoning through a novel graphical model incorporating latent thinking processes and evaluation signals. Within this framework, we introduce the Bootstrapping Reinforced Thinking Process (BRiTE) algorithm, which works in two steps. First, it generates high-quality rationales by approximating the optimal thinking process through reinforcement learning, using a novel reward shaping mechanism. Second, it enhances the base LLM by maximizing the joint probability of rationale generation with respect to the model's parameters. Theoretically, we demonstrate BRiTE's convergence at a rate of $1/T$ with $T$ representing the number of iterations. Empirical evaluations on math and coding benchmarks demonstrate that our approach consistently improves performance across different base models without requiring human-annotated thinking processes. In addition, BRiTE demonstrates superior performance compared to existing algorithms that bootstrap thinking processes use alternative methods such as rejection sampling, and can even match or exceed the results achieved through supervised fine-tuning with human-annotated data.

• The paper introduces a unified probabilistic framework that integrates latent thinking processes with reinforcement learning to improve LLM reasoning.
• It presents the BRiTE algorithm which alternates between RL policy updates and fine-tuning, achieving notable gains on benchmarks like GSM8K and HumanEval.
• Results demonstrate that RL-generated rationales can outperform manual annotations, providing a scalable approach to enhance language model reasoning.

Formalizing and Enhancing LLM Reasoning via Bootstrapping Reinforced Thinking: An Analysis of BRiTE

The paper "BRiTE: Bootstrapping Reinforced Thinking Process to Enhance LLM Reasoning" (2501.18858) introduces a unified probabilistic framework that systematically models the reasoning workflow in LLMs, explicitly incorporating latent thinking processes and evaluation signals. This framework motivates the BRiTE algorithm, which leverages reinforcement learning (RL) to automate the generation of high-quality rationales (thinking processes), and integrates them into LLM post-training to strengthen reasoning abilities.

Probabilistic Graphical Modeling of LLM Reasoning

The authors extend traditional LLM modeling, which treats the mapping from prompts $X$ to outputs $Y$ as direct, by introducing two key latent elements:

• $Z$, representing the latent thinking process (e.g., explicit CoT rationales),
• $O$, the evaluation signal indicating the quality or correctness of a generated rationale/answer pair.

This modeling factorizes reasoning as:

$$P(z, y, o \mid x, \theta) = P(z \mid x, \theta) \cdot P(y \mid x, z, \theta) \cdot P(o \mid x, z, y)$$

where $\theta$ are the LLM parameters.

By shifting the focus to maximizing the joint likelihood of producing both high-quality rationales and correct answers, conditioned on evaluation, the formulation recasts the inherently intractable posterior over latent rationales as a core learning target.

The BRiTE Algorithm: EM-Generalized RL Framework

BRiTE generalizes the Expectation-Maximization (EM) paradigm. It alternates between:

1. E-step (ψ-update): Learning a policy $\mathbb{Q}$ to sample high-quality thought processes $Z$ (with, optionally, $Y$ and $O$ fixed), proportional to the current model's posterior over $Z$ conditioned on observed success signals. Notably, BRiTE solves this using reinforcement learning—specifically entropy-regularized MDPs and PPO—rather than standard sampling or rejection techniques.
2. M-step (θ-update): Fine-tuning the model to maximize the likelihood of generating these bootstrapped rationales and answers.

The central innovation is to use RL-driven reward shaping to overcome the intractability of sampling or marginalizing over reasoning processes in real-world settings. Theoretical guarantees, including a convergence rate of $1/T$ under mild assumptions, position BRiTE on rigorous analytical footing. The framework unifies and subsumes SFT, RLHF (e.g., PPO, DPO), and various rejection sampling-based fine-tuning algorithms.

Empirical Validation and Key Findings

Empirical evaluations were conducted across math and coding benchmarks (GSM8K, MATH, HumanEval, BigCodeBench) and multiple open-source LLMs (Gemma, Llama, Mistral, DeepSeek). The comparison centered on three baselines:

• Supervised Fine-tuning (SFT): Fine-tuning with human-annotated thinking processes,
• Rejection Sampling (RS): Filtering auto-generated rationales via answer correctness before fine-tuning,
• RLHF Iterative DPO: Preference-based direct policy optimization.

Key results demonstrate: 1. Substantial improvement over rejection sampling approaches: For instance, BRiTE improved GSM8K accuracy on Gemma-1.1-7B-it by 1–10 points over RS, consistently across models and tasks. 2. RL-generated rationales matching or surpassing SFT on human-labeled CoT data: In some settings, BRiTE fine-tuned models exceeded the performance of those using expensive human-annotated rationales, without requiring any such labels. 3. Advancement in RLHF settings: BRiTE, when combined with iterative DPO, provided further gains over preference-learning-only baselines for reasoning-intensive benchmarks. 4. Effective generalization to code generation: On HumanEval and BigCodeBench, BRiTE enhanced LLM performance (e.g., boosting pass@1 rates) even when filtering/test mechanisms were impractical for baseline methods.

Implementation Considerations

Implementing BRiTE in practice involves:

• Integrating RL (e.g., PPO) frameworks for policy optimization over rationale sequences. Careful reward shaping—using log-probabilities from the existing model, with regularization—ensures tractable and stable RL convergence.
• Constructing data pipelines to generate and verify candidate rationales, with minimal reliance on human-labeled solutions.
• Efficient parameter-efficient fine-tuning (e.g., with LoRA) to avoid prohibitive compute requirements in large models.
• Modular design, enabling insertion into existing SFT or RLHF training curricula, particularly when direct human annotation or gold-standard CoT data is unavailable or costly.

A schematic of the process is as follows:

+-----------------+ Prompt (x) -->| RL-based |---[Generate rationale z]--> | CoT Generator | | +-----------------+ | | V RL (PPO; reward: log P(z,y*|x)) | | +-----------------+ | (z, x) ------>| Fine-Tuning |<--------------------------+ | (SFT/RLHF) | +-----------------+

Theoretical and Practical Implications

This research demonstrates that reinforcement learning on reasoning trajectories—rather than answer-only or preference-based fine-tuning—enables LLMs to internalize higher-quality, more robust implicit thought processes. The empirical evidence that RL-generated rationales can match or even outperform human CoT labels challenges prevailing assumptions about the necessity of extensive manual annotation for reasoning-intensive domains.

Theoretically, positioning LLM post-training as graphical latent variable modeling, with algorithmic ties to EM and modern policy optimization, provides a scaffold for extending reasoning-focused LLM research. This opens the door to:

• Richer model-based exploration of internal reasoning,
• Integration of verifier/reward models for complex downstream tasks,
• Frameworks for automated curriculum construction in reasoning data generation.

Limitations and Future Directions

While the bridging of RL and latent rationale generation demonstrates notable empirical gains, some aspects warrant further paper:

• The sensitivity of RL policy optimization dynamics (e.g., reward shaping, exploration in rationale space) to model initialization and task difficulty,
• Generalization to tasks with weaker or less binary evaluation signals,
• Scalability to extremely long reasoning chains or open-ended tasks in domains such as scientific discovery or legal reasoning.

Future research may explore tighter integration of verifier models, richer latent structure beyond chain-of-thought (e.g., trees/graphs of reasoning, as in Tree-of-Thought), and joint optimization of rationale and answer evaluation in both supervised and preference-based settings.

Overall, BRiTE substantiates the claim—both theoretically and empirically—that reinforcement learning over reasoning processes offers a scalable, general approach to enhancing the reasoning capabilities of LLMs, reducing reliance on human annotation, and laying the foundation for next-generation automated reasoners.