DLER: Optimizing Length Penalties in LLMs

• DLER is a framework that integrates mathematically principled, adaptive length penalties into reinforcement learning to balance brevity and accuracy.
• It employs techniques like Lagrangian duality, adaptive scheduling, and constrained optimization to reduce generation costs while maintaining model correctness.
• Empirical results demonstrate that DLER improves intelligence per token by dynamically adapting penalties based on prompt complexity.

Doing Length pEnalty Right (DLER) refers to a rigorous, empirically validated framework for optimizing reasoning efficiency in large language and reasoning models (LLMs/LRMs) by integrating mathematically principled length penalties into reinforcement learning (RL) training objectives. DLER contrasts with naïve, static, or one-size-fits-all length control, prioritizing a stable trade-off between output brevity and accuracy, and optimizing "intelligence per token"—accuracy normalized by generation length. State-of-the-art DLER approaches synthesize adaptive policy-gradient reward shaping, per-instance or per-batch penalty adaptation, and constrained optimization formalism to achieve substantial reductions in generation costs while often improving or retaining model correctness across a wide range of problem difficulties.

1. Motivation and Problem Formulation

Modern LLMs routinely employ multi-step, verbose chain-of-thought reasoning to maximize accuracy on complex tasks. However, this approach leads to unnecessarily long responses (high token counts) on simple or well-understood prompts, imposing significant computational costs and latency. Uniform, fixed-length penalties or truncation can induce "accuracy collapse" or inefficient use of compute budgets, as these penalties (i) do not adapt as model abilities evolve; (ii) fail to distinguish between "easy" and "hard" problems; and (iii) can over- or under-penalize depending on local context (Li et al., 25 Dec 2025, Liu et al., 16 Oct 2025, Su et al., 23 May 2025, Xiang et al., 5 Jun 2025, Li et al., 25 Jun 2025, Ling et al., 12 Jun 2025, Yu et al., 12 Feb 2026).

DLER methods reconceptualize length control as a constrained optimization problem:

$$\text{maximize}_{\theta}\;\;\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot|x)}[r(x,y)]$$

subject to

$\mathbb{E}[L(y)] \leq L_t$

where $r(x, y)$ is a correctness-based reward (e.g., exact match, pass@1), $L(y)$ is the response length, and $L_t$ is a target budget. This framework underpins adaptive length penalties, reward shaping, and dynamic constraint satisfaction (Li et al., 25 Dec 2025, Liu et al., 16 Oct 2025, Su et al., 23 May 2025, Xiang et al., 5 Jun 2025, Li et al., 25 Jun 2025, Ling et al., 12 Jun 2025, Yu et al., 12 Feb 2026).

2. Key DLER Methodologies

Multiple instantiations of DLER have been developed, unified by their emphasis on adaptation, policy-driven reevaluation, and integration of length as an information-coupled signal.

a) Lagrangian Duality and Primal–Dual Adaptive Penalties

Leash (Li et al., 25 Dec 2025) embodies DLER via a Lagrangian constrained RL formalism, introducing a nonnegative dual variable (Lagrange multiplier) $\lambda$:

$$\mathcal{L}(\theta, \lambda) = \mathbb{E}[r(x, y)] - \lambda \,\mathbb{E} \left[ \frac{L(y)}{L_t} - 1 \right]$$

Primal policy updates maximize this composite objective, while dual updates adjust $\lambda$ online according to the normalized average length violation:

$$\lambda \leftarrow \mathrm{Clip}_{[0,1]}\left[\lambda + \alpha_\lambda \cdot \widehat{J_P} \right]$$

where $\widehat{J_P}$ estimates constraint violation over the batch. A one-sided penalty $\mathbb{E}[L(y)] \leq L_t$0 ensures that brevity receives no extra reward, stabilizing training.

b) Adaptive Direct Length Penalty (A-DLP) and Powered Length Penalty

In A-DLP (Su et al., 23 May 2025), the length penalty coefficient $\mathbb{E}[L(y)] \leq L_t$1 is dynamically scheduled with respect to observed accuracy:

$\mathbb{E}[L(y)] \leq L_t$2

$\mathbb{E}[L(y)] \leq L_t$3

The "Powered Length Penalty" (Ling et al., 12 Jun 2025) variant further refines the reward:

$\mathbb{E}[L(y)] \leq L_t$4

where brevity is only rewarded when the answer is correct, mitigating collapse on difficult prompts.

c) Per-Instance and Accuracy-Aware Penalty Schedules

Several DLER recipes deploy prompt-specific adjustments based on empirical difficulty:

• Adaptive Length Penalty (ALP) (Xiang et al., 5 Jun 2025): $\mathbb{E}[L(y)] \leq L_t$5, where $\mathbb{E}[L(y)] \leq L_t$6 is the rolling solve rate for prompt $\mathbb{E}[L(y)] \leq L_t$7. Easy prompts attract large penalties, strongly discouraging excess tokens; hard ones tolerate longer reasoning.
• AALC (Li et al., 25 Jun 2025): Length penalty is "delayed" until validation accuracy reaches a moving target $\mathbb{E}[L(y)] \leq L_t$8; penalty scales smoothly as

$\mathbb{E}[L(y)] \leq L_t$9

ensuring correctness is prioritized before brevity.

d) Coordinated Reflection and Length Penalties

ARLCP (Yu et al., 12 Feb 2026) extends DLER to penalize both unnecessary reflection (self-questioning, circular reasoning) and overall length, with adaptive weights co-allocated from a fixed penalty budget $r(x, y)$0. Penalty strength is determined dynamically by the observed frequency of reflection tokens and normalized using batch statistics.

3. Optimization Challenges and Algorithmic Innovations

DLER effectiveness relies on addressing core RL optimization pathologies unique to large-scale, sparse-reward, truncation-heavy environments (Liu et al., 16 Oct 2025):

• Advantage Bias: Aggressive truncation induces high variance and bias in advantage estimation. DLER employs batch-wise normalization (instead of group-wise) to mitigate this effect.
• Entropy Collapse: Uniform PPO clipping collapses output diversity, especially at low-probability tokens; DLER adapts upper/lower clipping asymmetrically to preserve exploration and output diversity during brevity-oriented training (Liu et al., 16 Oct 2025).
• Curriculumized Sampling: Dynamic batch filtering, discarding all-positive/zero reward batches, avoids miscalibrated gradients on trivial or impossible prompts and ensures a stable, progressive learning curriculum.

4. Empirical Results and Comparative Performance

DLER variants have been extensively validated across deep LLM architectures (DeepSeek-R1, Qwen, Qwen2.5), benchmarks (GSM8K, AIME24/25, MATH500, AMC23), and settings (IID, OOD problems) (Li et al., 25 Dec 2025, Su et al., 23 May 2025, Liu et al., 16 Oct 2025, Xiang et al., 5 Jun 2025, Li et al., 25 Jun 2025, Ling et al., 12 Jun 2025, Yu et al., 12 Feb 2026). Key results include:

Model & Setting	Baseline Accuracy / Tokens	DLER Variant / Tokens (↓) / Accuracy (↑)	Efficiency Notes
DeepSeek-1.5B (AIME)	33.1% / 15,727	Leash: 33.9% / 5,873 (–62.7%)	Adaptive λ best trade-off (Li et al., 25 Dec 2025)
DeepSeek-7B (AIME)	67.2% / 7,747	DLER: 69.4% / 2,405 (–69%)	+2.2 pp accuracy (Liu et al., 16 Oct 2025)
Qwen2.5-7B (GSM8K)	91.4% / 298	DLER: 90.3% / 182 (–39%)	PLP avoids harsh penalty (Ling et al., 12 Jun 2025)
DeepScaleR-1.5B	—	ALP: 50% token cut, steady accuracy	Hard–easy token allocation (Xiang et al., 5 Jun 2025)
Qwen2.5-Math + AALC	93.2% / 439	97.6% / 97 (–78%)	Penalty delayed for accuracy (Li et al., 25 Jun 2025)
Reflection-aware	—	ARLCP: up to 53% token cut, +5.8pp acc	Reflection & length decoupled (Yu et al., 12 Feb 2026)

DLER techniques universally improve the accuracy–efficiency Pareto frontier versus both uniform-length penalties and specialized baselines (Laser-DE, AdaptThink, etc.). Best practices emphasize per-prompt adaptation, dynamic penalty scheduling, and explicit constraint enforcement.

5. Best Practices and Implementation Guidelines

Observed across multiple DLER studies:

• Penalty strength modulation: Adaptive or per-prompt λ/α enables strong brevity push when safe (on easy or well-solved prompts), but relaxes pressure automatically to avoid sacrificing accuracy on difficult cases (Xiang et al., 5 Jun 2025, Su et al., 23 May 2025, Li et al., 25 Dec 2025).
• Constraint satisfaction: Lagrangian or moving-target scheduling ensures hard caps on expected length, mitigating regression in response length over time.
• Reward shaping: One-sided and clipped penalties avoid over-rewarding (or over-penalizing) excessively short outputs (Li et al., 25 Dec 2025, Liu et al., 16 Oct 2025).
• Training dynamics: Curriculumized sampling and batch normalization reduce instability and enable convergence without accuracy collapse (Liu et al., 16 Oct 2025, Li et al., 25 Jun 2025).
• Practicality: Most DLER variants incur minimal code overhead—modifying only the RL reward and a handful of bookkeeping lines.

6. Conceptual Advances and Impact

DLER moves the field beyond rigid, fixed-length controls or static brevity rewards, establishing that robust intelligence-per-token optimization is only possible with adaptive, context-sensitive, and theoretically grounded penalty integration. The approach is extensible to complexity-aware chain-of-thought compression, reflection control, and scalable to larger or instruction-tuned models (Yu et al., 12 Feb 2026, Liu et al., 16 Oct 2025, Li et al., 25 Jun 2025).

The core insight, repeatedly validated, is that robust RL optimization and adaptive reward scaling—not simply the form of the penalty—determine the achievable trade-off frontier. In particular, hard truncation combined with stable advantage estimation suffices for state-of-the-art results; more intricate penalty functions move models along, but not beyond, this efficiency frontier (Liu et al., 16 Oct 2025).

7. Open Directions and Limitations

DLER research identifies several avenues for further work:

• Scaling adaptive schemes to ultra-large models and multi-modal settings (Su et al., 23 May 2025, Li et al., 25 Jun 2025).
• Per-instance penalty estimation: More principled integration of problem or prompt complexity signals (Xiang et al., 5 Jun 2025, Yu et al., 12 Feb 2026).
• Adaptive reflection–length budget allocation: Jointly optimizing reasoning structure and conciseness (Yu et al., 12 Feb 2026).
• Improving interpretability: Striking a balance between compression and the loss of explanatory detail (Li et al., 25 Jun 2025).
• RL algorithm engineering: Orchestrating batch norm, clipping, and advantage estimation for extreme truncation regimes (Liu et al., 16 Oct 2025).

A plausible implication is that DLER-like strategies could serve as a universal "efficiency shaper" for automated reasoning systems in both academic and high-throughput industrial deployments, provided model-specific calibration and fast per-task adaptation are available.