Direct Reasoning Optimization (DRO)

Direct Reasoning Optimization (DRO) is a reinforcement learning (RL) framework designed to fine-tune LLMs for open-ended, particularly long-form, reasoning tasks, where generic, verifiable reward signals are unavailable. DRO is motivated by the limitations of RL with Verifiable Rewards (RLVR), which relies on scalable, outcome-based reward signals effective for structured tasks (such as mathematics and programming) but is hard to apply for subjective or open-domain settings like document revision or analytic summarization. DRO introduces the Reasoning Reflection Reward (R³), a self-supervised, token-level reward mechanism that attributes the value of reasoning traces to model-generated intermediate steps, and leverages this for a self-contained, reward-model-free RL pipeline for LLMs.

1. Rationale and Theoretical Foundations

DRO is constructed to bridge the gap faced by RLVR when scaling reinforcement learning to open-ended reasoning assessments, for which no automatic verifier or objective correctness signal exists. In traditional RLVR, reward signals are directly tied to externally verifiable outcomes (e.g., correctness, test accuracy), allowing for robust, hack-resistant training. For open-ended tasks, such signals are either missing or unreliable, making metric gaming (reward hacking) and overfitting to brittle similarity measures a central concern.

DRO employs internal model uncertainty and sensitivity to intermediate reasoning to produce an interpretable, informative reward. Instead of relying on external reference models, LLM-as-a-judge schemes, or automatic evaluators, DRO uses the model's own predictions to generate the Reasoning Reflection Reward (R³) at the token level. This enables optimization of the LLM's reasoning ability in a way that is consistent, scalable, and immune to external model hallucinations.

2. Reasoning Reflection Reward (R³): Definition and Computation

At the core of DRO is R³, which attributes performance to the model’s own internal reasoning by focusing attention on the subset of output tokens whose probabilities are sensitive to the prior reasoning path. The key steps are:

• For a given input prompt $q$, generate multiple chain-of-thought (CoT) reasoning traces $\{\hat{c}_i\}$.
• For each reasoning sample $\hat{c}_i$, compute the log-probabilities assigned to each token $y_j$ in the reference output $y$: $\log \pi(y_j | q, \hat{c}_i, y_{<j})$.
• For every token $y_j$, across reasoning traces, compute the standard deviation $\sigma_j$ of these log-probabilities.
• Identify "reasoning-reflective" tokens as those with large $\sigma_j$ (high likelihood sensitivity to reasoning path).
• Aggregate the R³ for each reasoning trace and output as:

$$R^{(i)} = \sum_{j=1}^{|y|} w_{\Delta}(\sigma_j) \log \pi(y_j | q, \hat{c}_i, y_{<j})$$

where $w_{\Delta}(\sigma_j)$ weights by sensitivity. An additional correction subtracts model certainty in masked-reasoning traces to prevent superficial reward assignment for uninformative outputs:

$$R^{(i)} = \sum_{j=1}^{|y|} w_{\Delta}(\sigma_j) \left[ \log \pi(y_j | q, \hat{c}_i, y_{<j}) - \log \pi(y_j | q, c_{\text{masked}}, y_{<j}) \right]$$

This construction ensures that reward is attributed only to those reasoning steps that causally contribute to uncertainty reduction on the key output tokens, not merely to common context or shallow sequence artifacts.

3. Self-Contained, Reward Model-Free Training

DRO’s closed feedback loop is a distinguishing feature. Unlike methods that require additional verifiers or external reward models, DRO uses the same base LLM architecture (possibly with synchronized weights) for reference, actor, and reward computation. The reward signal is derived entirely within the model under optimization, by marginalizing the uncertainty and token attribution scores over the sampled reasoning traces. This approach addresses concerns of generator-discriminator gaps, aligns the training and evaluation metrics, and simplifies pipeline deployment.

The self-contained nature of R³ estimation also means there is no dependence on prompt engineering, external reward model biases, or empirical judge variability. As a result, DRO scales seamlessly to new domains and languages where reference judges or reward models may be missing.

4. Dynamic Data Filtering for Efficient RL

DRO incorporates a dynamic data filtering mechanism using R³ to prioritize informative training examples and reduce computational cost. Specifically, for each example, multiple reasoning traces are sampled and their R³ scores inspected:

• Examples are excluded if even the best reasoning trace yields negligible probability for key tokens (indicating the problem is out-of-domain or too hard for the current model).
• Examples with low variance in R³ across reasoning samples are deprioritized, as they present little opportunity for learning (being either trivial or redundant).
• Only prompts with significant reasoning trace sensitivity and tractable challenge are retained for RL.

This approach reduces computational demands by up to 45% and accelerates convergence, as shown in empirical studies.

5. Empirical Evaluation: Open-Ended and Structured Tasks

DRO is evaluated on both open-ended and structured datasets:

ParaRev (long-form paragraph revision): Models must rewrite scientific paragraphs to better align with reviewer feedback. DRO-trained LLMs (Qwen-14B variants) achieve 51.8% (GPT-4o judge) and 58.8% (Claude Sonnet judge) win rates—exceeding the strong baselines and even surpassing GPT-4o on this difficult open-ended task. Traditional reward hacking (e.g., via ROUGE) leads to unfaithful revisions, which DRO robustly avoids. Length and structure matching remain faithful to the reference, indicating high alignment.

FinQA (math-oriented QA): DRO achieves 67–82% Pass@k, on par or ahead of correctness-verifier RL baselines, despite having no explicit correctness signals. Baselines using aggregate token certainty (rather than R³ attribution) underperform, showing that global certainty is insufficient to drive meaningful reasoning optimization.

These results indicate that reasoning-reflective token attribution is effective for both subjective, open-ended tasks and structured, verifiable ones.

6. Broader Implications and Applicability

DRO’s innovations provide several key advantages and implications:

• No dependency on external reward models or judges: Eliminates both cost and bias, facilitating deployment where gold references are available but automatic evaluation is not.
• Robust to metric gaming and reward hacking: By emphasizing tokens causally connected to reasoning pathways, DRO is less susceptible to superficial improvements (e.g., copying or shallow paraphrasing).
• Generalizes to unstructured, long-form domains: The same framework applies to summarization, editing, analytic report writing, and other ambiguous comprehension challenges, as well as domains with verifiable outcomes (e.g., math, programming).
• Reduces optimization cost: Dynamic R³-based filtering systematically concentrates training on learnable, non-trivial data subsets.
• Serves as a blueprint for future RL-based LLM optimization: The combination of internal uncertainty exploitation for reward shaping and domain-agnostic, reference-based scaling suggests broad utility for next-generation model alignment strategies.

In summary, Direct Reasoning Optimization (DRO) enables LLMs to self-improve their chain-of-thought reasoning on both open-ended and structured tasks, using only internal uncertainty and attribution mechanisms. It offers a scalable path forward for RL-driven reasoning refinement in domains previously inaccessible to outcome-verifiable RL, supporting the development of truly robust and insightful AI systems.