Dynamic Reasoning Quota Allocation (DRQA)

• Dynamic Reasoning Quota Allocation (DRQA) is a method that adaptively controls reasoning length in language models by balancing answer correctness and verbosity.
• It leverages batch-induced resource competition and reinforcement learning with preference data to guide the generation and early termination of reasoning chains.
• Experimental results demonstrate over 30% token reduction and a 1.5% average accuracy gain on diverse benchmarks, highlighting its efficiency and efficacy.

Searching arXiv for the cited DRQA and related optimization work. Dynamic Reasoning Quota Allocation (DRQA) is a method for controlling overthinking in reasoning LLMs (RLLMs), introduced in the context of models such as OpenAI-O3 and DeepSeek-R1 (Yan et al., 25 Aug 2025). The central problem is that RLLMs often produce unnecessarily lengthy reasoning chains even for simple questions, which increases token consumption and computational inefficiency without necessarily improving answer quality. DRQA transfers the emergent efficiency observed in batch processing—where implicit resource competition compresses reasoning for easier problems—into single-question inference by learning to allocate reasoning resources adaptively. In the original formulation, DRQA combines batch-generated preference data with reinforcement learning so that the model internalizes a preference for responses that are both accurate and concise, generating shorter answers for simple questions while retaining sufficient reasoning depth for harder ones (Yan et al., 25 Aug 2025).

1. Problem formulation and objective

DRQA is formulated around an explicit trade-off between answer correctness and reasoning length (Yan et al., 25 Aug 2025). Let $x$ denote an input question, with associated latent difficulty $d\in\mathcal D$. A reasoning chain is written as $\pi=(z_1,z_2,\dots,z_T)$, where $T$ is the number of reasoning steps or tokens generated, and $\pi_{1:t}=(z_1,\dots,z_t)$ denotes the prefix up to step $t$. The model’s stochastic policy for generating the next token is $\pi_\theta(z_{t+1}\mid x,\pi_{1:t})$. Correctness is represented by $\mathrm{Acc}(\pi,x)\in\{0,1\}$, and token usage is $U(\pi)=|\pi|=T$. With a penalty coefficient $\lambda>0$, the utility of a reasoning chain is defined as

$d\in\mathcal D$0

The learning objective is to maximize expected utility under the data distribution: $d\in\mathcal D$1 An equivalent view presented in the formulation is that DRQA learns a mapping $d\in\mathcal D$2 that prescribes a reasoning-token quota adapted to problem difficulty. In this sense, “quota allocation” is not a fixed global cap, but an instance-adaptive control mechanism. A common misconception is that DRQA is merely a length penalty; the stated objective instead couples correctness and brevity at the level of expected utility, and the method is explicitly described as adaptive rather than static (Yan et al., 25 Aug 2025).

2. Preference data from batch-induced resource competition

The empirical starting point of DRQA is the observation that when multiple questions $d\in\mathcal D$3 are processed in a single batch, the model exhibits “resource competition pressure” and implicitly compresses reasoning for easier problems, yielding shorter chains without hurting accuracy (Yan et al., 25 Aug 2025). DRQA harvests this phenomenon to construct a preference dataset.

For each sampled question $d\in\mathcal D$4, two reasoning traces are generated: a “vanilla” chain $d\in\mathcal D$5 by prompting alone, and a “batch” chain $d\in\mathcal D$6 by including $d\in\mathcal D$7 in a batch of size $d\in\mathcal D$8. These traces are then labeled according to correctness and conciseness using three evaluation choices: Option A, “correct but verbose,” if $d\in\mathcal D$9 is correct; Option B, “correct and concise,” if $\pi=(z_1,z_2,\dots,z_T)$0 is correct; and Option C, “incorrect,” if the chain is wrong (Yan et al., 25 Aug 2025).

The resulting data support pairwise preference modeling. Preference pairs $\pi=(z_1,z_2,\dots,z_T)$1 are scored by a learned scoring function $\pi=(z_1,z_2,\dots,z_T)$2, with preference probability

$\pi=(z_1,z_2,\dots,z_T)$3

where $\pi=(z_1,z_2,\dots,z_T)$4. These preferences are intended to capture the implicit trade-off made by the batch model between correctness and brevity. This suggests that the batch setting is treated not merely as an efficiency artifact, but as a source of supervision for learning concise reasoning behavior in the single-instance regime.

3. Reinforcement-learning formulation

DRQA casts quota allocation as a reinforcement-learning problem in which the agent learns both to generate reasoning chains and to decide when to stop (Yan et al., 25 Aug 2025). The Markov decision process is defined with state $\pi=(z_1,z_2,\dots,z_T)$5, action space $\pi=(z_1,z_2,\dots,z_T)$6, and transitions that either sample a new token $\pi=(z_1,z_2,\dots,z_T)$7 or terminate the episode. Rewards are zero for generation steps, and at stopping time the terminal reward is

$\pi=(z_1,z_2,\dots,z_T)$8

The policy objective is the expected cumulative reward

$\pi=(z_1,z_2,\dots,z_T)$9

with policy-gradient expression

$T$0

where $T$1 is the episode return (Yan et al., 25 Aug 2025).

In practice, DRQA uses Group Relative Policy Optimization (GRPO) to inject preference data as a supervised “advantage” signal. Let $T$2 denote the evaluation choices and $T$3 the correct label. The GRPO-style loss is given as

$T$4

where $T$5 if $T$6 and negative otherwise, and $T$7 is a KL-penalty coefficient. The pseudocode sketch initializes from a pretrained model $T$8, computes $T$9 on batches of preference examples, and updates parameters by gradient descent. The formal role of GRPO in DRQA is therefore not generic RLHF, but a preference-conditioned policy optimization procedure tailored to the A/B/C evaluation space.

4. Adaptive inference and stopping behavior

At test time, DRQA uses the learned policy $\pi_{1:t}=(z_1,\dots,z_t)$0 to decide on the fly whether to continue reasoning or to stop (Yan et al., 25 Aug 2025). The primary stopping criterion compares the log-odds of stopping versus continuing: $\pi_{1:t}=(z_1,\dots,z_t)$1 Reasoning terminates as soon as $\pi_{1:t}=(z_1,\dots,z_t)$2 for some threshold $\pi_{1:t}=(z_1,\dots,z_t)$3.

An equivalent formulation compares a value estimate for continuation against the immediate reward of stopping: $\pi_{1:t}=(z_1,\dots,z_t)$4 These criteria operationalize the learned quota as a sequential decision process rather than an externally imposed maximum length. The mechanism is therefore dynamic in two senses: the effective quota is instance-specific, and the stop decision is recomputed at each reasoning step. A plausible implication is that DRQA can terminate early on easy instances without preventing long trajectories on difficult ones, because the stopping condition is conditioned on the evolving state $\pi_{1:t}=(z_1,\dots,z_t)$5 rather than on a predetermined token budget.

5. Experimental evidence

The reported evaluation covers six benchmarks spanning elementary to graduate-level mathematical and scientific reasoning: GSM8K, MATH-500, AIME-2024, AIME-2025, AMC-2023, and GPQA-Diamond (Yan et al., 25 Aug 2025). The experiments use two distilled LLM sizes, 1.5 B and 7 B parameters, from DeepSeek-R1. Baselines include vanilla prompting, O1-Pruner, DAST, ShorterBetter, GRPO-only, GRPO+length-penalty, supervised fine-tuning (SFT), AdaptThink, AutoL2S, Dynasor-CoT, and DEER.

For the 1.5 B model, the key aggregated results show that Vanilla attains 53.95% overall accuracy with 10,124.1 average tokens; GRPO attains 55.66% with 9,787.1 tokens; SFT attains 49.82% with 12,784.4 tokens; and DRQA attains 55.69% with 6,801.7 tokens, corresponding to a token reduction of $\pi_{1:t}=(z_1,\dots,z_t)$6 relative to Vanilla (Yan et al., 25 Aug 2025). On GSM8K with the 1.5 B model, Vanilla achieves 84.67% accuracy with 1,928.96 tokens, whereas DRQA achieves 86.67% accuracy and 1,427.63 tokens, reported as $\pi_{1:t}=(z_1,\dots,z_t)$7 percentage points in accuracy and $\pi_{1:t}=(z_1,\dots,z_t)$8 in token usage. On MATH-500 with the 7 B model, Vanilla achieves 90.40% accuracy with 5,099.95 tokens, while DRQA achieves 91.40% accuracy with 3,902.74 tokens, reported as $\pi_{1:t}=(z_1,\dots,z_t)$9 percentage point in accuracy and $t$0 in token usage (Yan et al., 25 Aug 2025).

Across all six benchmarks and both model sizes, DRQA is reported to achieve an average accuracy gain of $t$1 percentage points and a token-usage reduction of $t$2, outperforming all prior baselines in the trade-off curve (Yan et al., 25 Aug 2025). The experimental pattern is therefore not limited to a single dataset or model scale. It also bears directly on a frequent concern about concise reasoning methods: the reported results do not present a pure efficiency–accuracy trade-off, but rather a regime in which shorter reasoning can coincide with maintained, and in many cases improved, answer accuracy.

6. Interpretation, limitations, and later generalizations

The discussion accompanying DRQA attributes its performance to internalizing the batch-inference “resource competition” signal (Yan et al., 25 Aug 2025). Rather than hand-crafting a static token budget or imposing a fixed length penalty, the method is said to learn to weigh correctness versus verbosity on a per-instance basis. The same discussion states that RL training with preference data avoids the catastrophic accuracy drops seen in pure supervised fine-tuning for conciseness. This addresses a second common misconception: concise reasoning control in DRQA is not presented as an SFT compression problem, but as a preference- and reward-mediated policy learning problem.

The limitations are explicitly stated. Current experiments focus on numerical and scientific reasoning; extending to open-ended or multi-modal tasks remains future work. Additional proposed directions include more fine-grained control at the level of rhetorical substeps, dynamic adjustment of $t$3 per instance, and meta-learning for $t$4 so that the model can self-tune its stopping threshold without manual calibration (Yan et al., 25 Aug 2025). These points delimit the scope of the original evidence and clarify that the method has not yet been established for open-ended generation.

A later paper, “How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization” (Fang et al., 22 Feb 2026), presents a broader optimization framework, DynaMO, and includes a section titled “From DynaMO to DRQA: A General Methodology.” In that exposition, uniform rollout allocation is proved suboptimal, a variance-minimizing allocation

$t$5

is derived, and Bernoulli variance $t$6 is used as a computable proxy for gradient informativeness. The same framework adds token-level advantage modulation to compensate for softmax-gradient attenuation on high-confidence correct actions and to stabilize excessive updates using entropy-change indicators (Fang et al., 22 Feb 2026). Its generalized DRQA steps are: maintain per-instance statistics $t$7, compute variance proxy $t$8, allocate sequence-level reasoning quota $t$9, generate rollouts and advantages, apply token-level modulation factors, and update the policy with a PPO-style optimizer.

This suggests that “DRQA” acquired a broader methodological meaning after the original single-question inference proposal. In the original paper, DRQA is the transfer of batch-induced efficiency into a single-question RL framework with adaptive stopping (Yan et al., 25 Aug 2025). In the later exposition, DRQA is described as a generalized methodology for any LLM reasoning task with verifiable or proxy rewards, integrating dynamic rollout allocation and token-level advantage modulation into existing RLHF or RLVR pipelines (Fang et al., 22 Feb 2026). The terminological continuity is clear, but the scope is wider in the later formulation.