Discrepancy-Constrained MDP in LLM Training

• DCMDP is a constrained reinforcement learning formulation that explicitly bounds train–inference mismatch in large language model post-training.
• It employs a token-level discrepancy metric with a tolerance region to balance free exploration and corrective penalties.
• The method uses Lagrangian relaxation and GRPO-based updates to achieve stable training and improved performance under heterogeneous training/inference setups.

Searching arXiv for the specified paper and closely related terms to ground the article in current metadata. Discrepancy-Constrained Markov Decision Process (DCMDP) is a constrained reinforcement learning formulation for LLM post-training under hidden train–inference mismatch. It was introduced in "Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancy" (Liu et al., 7 Jun 2026) to address the observation that RL training for LLMs can exhibit unpredictable sub-optimum performance or even training collapses when the training policy and the deployed inference policy are realized through disparate underlying engines and architecture. The central premise is that reward maximization should be coupled with an explicit constraint on train–inference behavior, so that the policy can explore freely inside an empirically identified safe region while being steered back when discrepancy exceeds a tolerance bound.

1. Problem setting and motivation

The motivating setting is RL-based LLM post-training, where a policy is optimized on decoding trajectories but is ultimately deployed through a potentially different inference stack. The paper attributes observed optimization failures to a hidden train-inference discrepancy, or mismatch, caused by differences in the underlying engines and architecture (Liu et al., 7 Jun 2026). In this setting, an RL objective based only on reward may be insufficient, because the policy can improve under the training backend while drifting away from the behavior realized at inference time.

A key empirical finding is that the training policy can actively self-correct such a discrepancy when provided with an appropriate learning signal. The same study further identifies a discrepancy tolerance region. Within this region, aggressively narrowing the discrepancy can suppress policy exploration and reduce learning efficiency; outside this region, reducing excessive discrepancy improves optimization consistency and raises the achievable local performance ceiling (Liu et al., 7 Jun 2026). This observation motivates a constrained optimization perspective rather than unconditional discrepancy minimization.

A plausible implication is that DCMDP is not merely a regularization heuristic. It treats train–inference mismatch as a first-class optimization constraint whose effect depends on whether the current policy remains inside or outside a safe operating region.

2. Formalization as a constrained MDP

DCMDP extends a standard Markov Decision Process $(S, A, P, R, \gamma)$ to include an explicit discrepancy constraint between the training policy $\pi_{\text{train}}$ and the deployed inference policy $\pi_{\text{infer}}$ (Liu et al., 7 Jun 2026). In the paper’s formulation:

• $S$ is the set of states, corresponding to LLM decoding contexts.
• $A$ is the action space, corresponding to next-token choices.
• $P(s' \mid s, a)$ is the transition kernel, described as deterministic in language modeling.
• $R(s,a)$ is the per-token reward from a verifier.
• $\gamma$ is the discount factor, often $\gamma = 1$ for finite-horizon generation.

The optimization objective is

$$\max_{\pi_{\text{train}}} \mathbb{E}_{\tau \sim \pi_{\text{train}}}\left[\sum_t R(s_t, a_t)\right]$$

subject to

$\pi_{\text{train}}$0

where the paper denotes the discrepancy tolerance bound by $\pi_{\text{train}}$1 (Liu et al., 7 Jun 2026). The reward term preserves the usual RL objective, while the constraint requires the training policy to remain close to the deployed inference policy under a black-box discrepancy metric.

This formalization places DCMDP within the broader class of constrained MDPs, but its defining feature is that the constraint is not expressed in terms of cost accumulated from the environment. Instead, it is expressed in terms of behavioral divergence between two policies instantiated through different execution backends. That distinction is central to the paper’s framing of train–inference mismatch as an optimization artifact of LLM post-training rather than a conventional environment-side safety cost.

3. Black-box discrepancy metric and the tolerance region

The discrepancy measure $\pi_{\text{train}}$2 is defined at token level and aggregated at sequence level (Liu et al., 7 Jun 2026). For each decoded token $\pi_{\text{train}}$3 in a trajectory $\pi_{\text{train}}$4 of length $\pi_{\text{train}}$5,

$\pi_{\text{train}}$6

and the per-token discrepancy is

$\pi_{\text{train}}$7

The sequence-level metric is

$\pi_{\text{train}}$8

This is explicitly a black-box discrepancy metric. The formulation does not require structural access to the source of mismatch; it only requires token probabilities under the training and inference policies on the same decoded sequences.

The tolerance region is an empirical component of the framework. The paper reports that small discrepancies below $\pi_{\text{train}}$9 do not harm learning, so penalties are activated only when $\pi_{\text{infer}}$0 (Liu et al., 7 Jun 2026). The bound $\pi_{\text{infer}}$1 is chosen by measuring the natural train–inference gap on a held-out set, with example values of approximately $\pi_{\text{infer}}$2 for Qwen-3-8B and approximately $\pi_{\text{infer}}$3 for Qwen-3-30B-A3B.

A common misconception would be that train–inference discrepancy should always be minimized as aggressively as possible. The reported ablations do not support that view. The claimed behavior is explicitly non-monotonic: too little tolerance leads to over-regularization, whereas too much tolerance deactivates the constraint and reduces the method to vanilla GRPO.

4. Lagrangian relaxation and saddle-point optimization

To adaptively balance reward improvement and discrepancy control, the paper introduces a Lagrangian relaxation mechanism (Liu et al., 7 Jun 2026). With policy parameters $\pi_{\text{infer}}$4, the Lagrangian is

$\pi_{\text{infer}}$5

where

$\pi_{\text{infer}}$6

The resulting optimization is written in min–max form as

$\pi_{\text{infer}}$7

In the paper’s interpretation, the primal update maximizes reward while paying a $\pi_{\text{infer}}$8-scaled penalty for discrepancy violations, and the dual update chooses $\pi_{\text{infer}}$9 to enforce the constraint $S$0 in expectation (Liu et al., 7 Jun 2026). The multiplier update is

$S$1

where $S$2 is the batch-average discrepancy and $S$3 is the dual learning rate.

This mechanism operationalizes the tolerance-region finding. When discrepancy remains within the allowed region, the penalty can stay weak, permitting exploration. When discrepancy exceeds the bound, the dual variable increases and strengthens the corrective signal. The paper characterizes this as stable dual-objective optimization (Liu et al., 7 Jun 2026).

5. Algorithmic realization in DC-GRPO

The implementation described in the paper is DC-GRPO, which incorporates the discrepancy constraint into a GRPO-style training loop (Liu et al., 7 Jun 2026). At iteration $S$4, the procedure is summarized as follows:

1. Roll out under inference policy $S$5 to sample $S$6 responses per prompt and record $S$7 for every token.
2. Recompute token probabilities $S$8 with the high-fidelity training backend on the same sequences.
3. Compute token discrepancies $S$9.
4. Compute standard group-relative advantage $A$0 from reward shaping.
5. Form the penalized, iteration-specific advantage

$A$1

1. Perform the PPO/GRPO clipped surrogate step using $A$2 in place of $A$3.
2. Update $A$4 by gradient ascent on the constraint violation.

The paper emphasizes that this requires only one extra forward pass per token for $A$5 beyond the standard GRPO pipeline and introduces a scalar $A$6 with no extra network (Liu et al., 7 Jun 2026). In practical terms, the method augments the policy-gradient signal with a tokenwise penalty while leaving the overall RL optimization architecture largely intact.

6. Empirical validation and deployment implications

The empirical study evaluates Qwen-3-8B, a dense model, and Qwen-3-30B-A3B, a Mixture-of-Expert model, on six mathematical reasoning datasets: AIME24/25, AMC23, MATH-500, Minerva, and Olympiad (Liu et al., 7 Jun 2026). The reported averages are summarized below.

Setting	Qwen-3-8B	Qwen-3-30B-A3B
GRPO avg, BF16 training + BF16 inference	$A$7	$A$8
DC-GRPO avg, BF16 training + BF16 inference	$A$9	$P(s' \mid s, a)$0
GRPO avg, BF16 training $P(s' \mid s, a)$1 FP8 deployment	$P(s' \mid s, a)$2	$P(s' \mid s, a)$3
DC-GRPO avg, BF16 training $P(s' \mid s, a)$4 FP8 deployment	$P(s' \mid s, a)$5	$P(s' \mid s, a)$6

Under BF16 training and BF16 inference, DC-GRPO improves the reported average from approximately $P(s' \mid s, a)$7 to approximately $P(s' \mid s, a)$8 for Qwen-3-8B and from approximately $P(s' \mid s, a)$9 to approximately $R(s,a)$0 for Qwen-3-30B-A3B (Liu et al., 7 Jun 2026). Under heterogeneous training and deployment, with BF16 training and FP8 inference, GRPO performance degrades and collapses in some cases, whereas DC-GRPO preserves or even exceeds BF16-only performance according to the reported averages.

The heterogeneous setting is particularly important for the stated application regime. The paper argues that DCMDP enables a heterogeneous training paradigm in which LLMs are optimized in a high-fidelity training setup while being explicitly aligned for low-cost, resource-constrained inference deployment (Liu et al., 7 Jun 2026). This suggests a role for DCMDP in scenarios where deployment efficiency requires an inference stack that does not exactly match the training-time execution path.

7. Ablations, boundary cases, and conceptual significance

The discrepancy tolerance region is supported by ablation over the constraint bound $R(s,a)$1 (Liu et al., 7 Jun 2026). Three boundary cases are reported:

• When $R(s,a)$2 is too small, the policy is over-regularized, $R(s,a)$3 saturates at $R(s,a)$4, and reward gains are harmed.
• When $R(s,a)$5 is too large, the constraint is never active, $R(s,a)$6, and the method degenerates to vanilla GRPO.
• A middle-ground $R(s,a)$7 yields the best trade-off, confirmed by sweeping $R(s,a)$8.

These cases clarify that DCMDP is not equivalent to simply adding a fixed discrepancy penalty. Its behavior depends on a constrained optimization geometry in which the multiplier adjusts according to current violation. The paper’s take-away is that the method folds hidden train–inference mismatch into the RL objective and lets the policy self-correct only when it steps outside an empirically validated safe region (Liu et al., 7 Jun 2026).

In conceptual terms, DCMDP reframes LLM RL instability as a dual-objective problem: exploration for reward and fidelity to deployed inference behavior must be balanced rather than optimized independently. The paper’s evidence supports the claim that such balancing yields significantly more stable training and higher final performance, especially under heterogeneous training/inference infrastructures (Liu et al., 7 Jun 2026).