Hybrid Median-length Policy Optimization

• HMPO is a single-stage reinforcement learning framework that compresses chain-of-thought reasoning by using an adaptive median-based token budget combined with correctness-gated rewards.
• It employs a cosine-decay token reward mechanism and a multiplicative reward formulation to enforce a correctness-first, length-second hierarchy during policy optimization.
• Empirical results show 19–46% token compression across diverse tasks with less than a 1 percentage-point accuracy drop, confirming its efficiency.

Searching arXiv for the HMPO papers and closely related references to ground the article in current sources. Hybrid Median-length Policy Optimization (HMPO) is a single-stage reinforcement learning framework for compressing chain-of-thought (CoT) reasoning in LLMs while preserving answer correctness. It builds on Group Relative Policy Optimization (GRPO) and replaces manual or static length control with three coupled mechanisms: an adaptive median-based budget computed from successful rollouts, a cosine-decay token reward that applies only to correct trajectories under that budget, and a multiplicative reward formulation that enforces a correctness-first, length-second hierarchy. In the standalone formulation, HMPO is trained exclusively on mathematical data yet is reported to generalize across math, code, science, and instruction-following tasks; in the Mach-Mind-4-Flash pipeline, it serves as the final token-efficiency stage after multi-expert RL and Multi-Teacher On-Policy Distillation (MOPD) (Zheng et al., 1 Jun 2026, Team, 10 Jul 2026).

1. Definition, scope, and nomenclature

HMPO denotes Hybrid Median-length Policy Optimization, not the Hybrid Policy Optimization (HPO) method for hybrid discrete–continuous action spaces. The distinction matters because HPO addresses mixed action-space reinforcement learning via pathwise and score-function gradients, whereas HMPO addresses CoT compression in autoregressive LLMs (Alvo et al., 14 May 2026). HMPO is also distinct from HiPO, which trains a model to switch between Think-on and Think-off modes through a hybrid data pipeline and mode-aware rewards; HMPO instead operates through sequence-level reward shaping around rollout length and correctness (Deng et al., 28 Sep 2025).

Within the Mach-Mind-4-Flash report, HMPO is Stage (3) of a post-training pipeline whose earlier stages train domain-specific RL experts and fuse them into a generalist via MOPD. HMPO takes the MOPD-fused checkpoint as input and outputs the deployed model. In that setting, its stated purpose is to address “overthinking”: models with strong CoT reasoning often produce reasoning chains that are longer than necessary, inflating latency and serving cost without proportional gains in accuracy (Team, 10 Jul 2026).

The “hybrid” in HMPO refers to the combination of an adaptive, data-driven budget signal with on-policy GRPO optimization. A plausible implication is that HMPO belongs to a broader class of post-training methods that do not alter the base decoding interface, but instead change the learned policy so that shorter reasoning emerges under standard inference.

2. Core reward design

HMPO’s formal structure is centered on per-prompt groups of rollouts. Let $n_i$ denote the length in tokens of rollout $i$, and let $\mathcal{C}$ be the index set of correct rollouts within the sampled group. The adaptive budget is defined as

$b=\mathrm{median}(\{n_i\mid i\in\mathcal{C}\}).$

This budget uses only correct rollouts, which isolates the length signal from failed traces and makes the budget difficulty-aware: harder queries yield longer correct traces and therefore a looser budget, while easier queries yield shorter budgets. Both the standalone paper and the technical report emphasize that the budget re-estimates on-policy and self-tightens as the policy improves (Zheng et al., 1 Jun 2026).

The token reward is

$$R_{\text{token}}= \begin{cases} \min\!\left(1,\; \cos\!\left(\frac{\pi n}{2b}\right)+\lambda\right), & \text{if correct and } n<b,\ 0, & \text{otherwise.} \end{cases}$$

and the final rollout reward is

$$R_{\text{final}}=R_{\text{acc}}\cdot R_{\text{token}}.$$

This multiplicative composition gives the method its correctness-gated character. Incorrect traces receive zero reward regardless of length, and over-budget correct traces receive zero token reward. The papers describe this as a strict correctness-first hierarchy and argue that it prevents reward hacking on short-but-wrong outputs more effectively than additive penalties, which can inadvertently reward either verbose-but-correct or short-but-wrong behavior (Team, 10 Jul 2026).

The standalone HMPO paper additionally specifies a fallback when a prompt has no correct rollouts in its sampled group: set $b\leftarrow\infty$, so all token rewards and final rewards are zero for that group. This means no length gradient is provided when correctness is absent, a design choice that avoids extracting compression signal from uniformly failed samples (Zheng et al., 1 Jun 2026).

3. Optimization framework and training procedure

HMPO is implemented as an on-policy GRPO stage. For each query, the policy samples a group of rollouts; in the reported training configuration, the group size is $G=10$. Group-relative advantages are computed from the rollout rewards, and standard GRPO updates are applied at sequence level rather than through any new gradient estimator. The Mach-Mind-4-Flash report is explicit that HMPO introduces no teacher routing, no distillation objective, and no additional gradient estimator beyond GRPO; its only algorithmic change is the reward shaping defined above (Team, 10 Jul 2026).

The standalone HMPO paper provides a more implementation-oriented description. It defines a token-level importance ratio, a normalized group-relative advantage, and a clipped surrogate objective in the PPO/GRPO style. Reported hyperparameters include clip ratio $\epsilon=0.2$, group size $G=10$, KL coefficient $i$0 for both reward and loss, entropy coefficient $i$1, AdamW optimization with learning rate $i$2 and minimum learning rate $i$3, cosine schedule, and $i$4 warmup. Precision is bfloat16, and the implementation is built in VERL. The same paper states that HMPO is trained on approximately 6.5K high-quality mathematics problems, for a single RL epoch, with convergence within 40–60 steps and the best compression/accuracy trade-offs around steps 16–24 (Zheng et al., 1 Jun 2026).

At inference time, HMPO does not require runtime routing, truncation heuristics, or special prompts. The model is decoded normally under standard sampling or greedy settings, and the shorter CoT is expected to emerge from the learned reward shaping. This differentiates HMPO from decode-time early-exit or post-hoc candidate filtering schemes, since compression is moved into the policy itself rather than layered on top of generation (Team, 10 Jul 2026).

4. Empirical performance and ablations

Across the standalone HMPO experiments, the reported headline result is 19%–46% token compression with negligible accuracy degradation, obtained across dense and Mixture-of-Experts models from 9B to 122B parameters. The paper reports that the method is trained on math only but generalizes across code, science QA, and instruction following (Zheng et al., 1 Jun 2026).

Model	Overall compression	Average accuracy
Qwen3.5-9B	46%	82.80% average ($i$5 pp vs base)
Qwen3.5-35B-A3B	42%	85.52% average ($i$6 pp vs base)
Qwen3.5-122B-A10B	19%	88.14% average ($i$7 pp vs base)

The same paper provides cross-benchmark breakdowns. For Qwen3.5-9B, compression is reported as 32% on AIME’25, 37% on AIME’26, 61% on LiveCodeBench V6, 34% on GPQA, and 41% on IFEval. For Qwen3.5-35B-A3B, the corresponding figures are 34%, 37%, 49%, 30%, and 61%. For Qwen3.5-122B-A10B, they are 23%, 23%, 16%, 17%, and 16%. One reported detail is that the 35B model improves on IFEval despite overall compression, reaching 94.09% versus 92.79% for the base model (Zheng et al., 1 Jun 2026).

Ablations in the 9B setting are used to justify the three design components. The default setting $i$8, median budget, multiplicative reward) yields 82.80% accuracy and 46.0% compression. Lowering $i$9 to 0.4 reduces accuracy to 79.79% and compression to 41.6%, while increasing $\mathcal{C}$0 to 1.0 gives 81.15% accuracy and 42.5% compression. Replacing the adaptive budget with a fixed $\mathcal{C}$1 at $\mathcal{C}$2 yields 82.12% accuracy but only 12.4% compression, and replacing multiplicative composition with an additive form yields 80.83% accuracy and 41.7% compression. These results are presented as evidence that adaptive budgeting and correctness-gated reward composition are not incidental implementation choices but central to the reported Pareto trade-off (Zheng et al., 1 Jun 2026).

In the Mach-Mind-4-Flash report, HMPO is described as compressing reasoning chains by 19–46% with $\mathcal{C}$3 percentage-point accuracy loss and shifting the AIME’26 accuracy–token Pareto frontier up and left. The report states that the HMPO-trained model decisively outperforms Nemotron-3-Super-120B-A12B at much lower token cost and approaches Kimi-K2.5-1T-A32B at substantially lower token cost (Team, 10 Jul 2026).

5. Relation to alternative efficiency methods

HMPO is framed against several alternative approaches to reasoning compression. Static length budgets are described as inflexible because they ignore instance difficulty. Multi-stage pipelines based on distillation, supervised fine-tuning, and RL or iterative pruning are described as computationally expensive and brittle. Early-exit and truncation heuristics act at decode time and risk cutting off valid reasoning. Best-of-$\mathcal{C}$4 selection reduces serving quality only post hoc and does not teach the policy to be concise in expectation (Team, 10 Jul 2026).

The standalone HMPO study compares against AdaptThink, Thinkless, and ThinkPrune. AdaptThink is reported to achieve moderate compression but can inflate length in some settings, such as a +33% change on IFEval for the 35B model. Thinkless is described as a two-stage SFT+RL method prone to mode collapse, with catastrophic accuracy drops in some settings. ThinkPrune yields very high compression, reported as 64–72%, but with severe accuracy degradation. HMPO is therefore positioned as occupying a more favorable accuracy–compression frontier while remaining single-stage (Zheng et al., 1 Jun 2026).

Relative to HiPO, the distinction is architectural as well as algorithmic. HiPO relies on paired Think-on and Think-off data, a learned judge segment, mode-aware advantages, and bias adjustment to discourage overuse of verbose reasoning. HMPO does not introduce mode tokens or judge-based routing; instead, it compresses reasoning directly through correctness-gated length rewards on grouped rollouts. This suggests that the two methods address related deployment concerns—accuracy preservation under lower token budgets—but operate at different abstraction levels (Deng et al., 28 Sep 2025).

6. Constraints, limitations, and open questions

HMPO’s reported limitations are tied to the assumptions built into its reward signal. It is currently targeted at single-turn reasoning, and the extension of budget-aware compression to multi-turn agentic trajectories, where effort must be allocated across many tool-call turns, is identified as an open problem. The method also depends on a verifiable correctness signal: if the checker is inaccurate, learning degrades because correctness gating controls all downstream reward (Team, 10 Jul 2026).

A second limitation is structural. The reward is smooth only within the interval $\mathcal{C}$5 and drops to zero at $\mathcal{C}$6. The standalone paper explicitly notes the “hard zero outside budget,” and suggests that in some domains a taper beyond $\mathcal{C}$7 could be considered in future work. The same source also notes that no length gradient is available when a prompt yields no correct rollouts, so training quality depends on task mixtures that avoid too many degenerate groups (Zheng et al., 1 Jun 2026).

A third issue is source-dependent implementation detail. The Mach-Mind-4-Flash report describes HMPO as pure GRPO reward shaping and states that KL regularization and reverse-KL objectives are exclusive to MOPD, whereas the standalone HMPO paper reports low-variance KL coefficients of 0.001 for both reward and loss in its implementation. The shared point is narrower than the implementation differences: HMPO’s defining innovation is not a new regularizer or estimator, but the median-adaptive, correctness-gated reward construction itself (Team, 10 Jul 2026, Zheng et al., 1 Jun 2026).

Taken together, the available formulations present HMPO as a post-training method for making reasoning models concise without discarding the benefits of CoT. Its technical identity lies in replacing manually budgeted or decode-time compression with an on-policy reward that measures brevity only among successful trajectories and only after correctness has been satisfied.