---
title: Reward-Weighted Dynamic Thresholding
url: https://www.emergentmind.com/topics/reward-weighted-dynamic-thresholding
type: topic
---

# Reward-Weighted Dynamic Thresholding

Searching arXiv for the cited work and closely related thresholding/reward-optimization papers.
Reward-Weighted Dynamic Thresholding denotes a cluster of threshold-centered optimization procedures in which rewards, advantages, or scalar feedback are transformed before they are used for learning. In recent arXiv work, the phrase is used most explicitly for diffusion-based autonomous motion planning, where adaptively tuned thresholds convert non-differentiable safety and effectiveness metrics into shaped rewards that weight diffusion log-likelihood gradients [2507.12977]. Closely related thresholding mechanisms also appear in multi-reward policy optimization for LLM post-training, where rollout-level masks are applied to conflict-prone advantage vectors [2606.16771], in latent-threshold learning with censored feedback, where the threshold itself is the learned decision variable [2312.04653], and in visual generative alignment, where a global threshold replaces an intractable instance-specific KL baseline [2605.04653]. These formulations are not identical, but they share a common concern: direct optimization is hindered by sparsity, censoring, or interference, and thresholding is used to recover an informative training signal.

## 1. Scope and recurring structure

Across the cited literature, thresholding enters the learning problem in several distinct ways.

| Setting | Thresholded quantity | Function |
|---|---|---|
| Diffusion motion planning | Per-metric threshold $\epsilon_m$ | Shapes dense rewards and balances positives near $B/2$ |
| GD$^2$PO | Conflict threshold $\tau$ | Masks rollouts with severe reward-wise disagreement |
| Latent-value learning | Decision threshold $\gamma$ | Maximizes censored utility $U(\gamma)$ |
| TGO | Global score threshold $\tau$ | Forms pseudo-labels from unpaired scalar scores |

In the motion-planning formulation, the threshold is explicitly adaptive inside training: each $\epsilon_m$ is updated so that the number of positives for metric $m$ is approximately half the batch, thereby avoiding all-zero or all-one rewards [2507.12977]. In GD$^2$PO, the threshold is applied to a rollout conflict metric, either a hard sign-consistency indicator or an SNR-based coherence score, in order to suppress advantage cancellation across reward dimensions [2606.16771]. In the latent-threshold problem, the learner repeatedly queries a threshold and receives censored rewards of the form $g(\gamma,v)\mathbf 1_{v\ge \gamma}$, so the threshold is the central object of estimation [2312.04653]. In Threshold-Guided Optimization, a global percentile-based threshold turns scalar reward scores into binary pseudo-labels and can be recomputed once per epoch or per batch [2605.04653].

A plausible implication is that “reward-weighted dynamic thresholding” is best treated as a methodological pattern rather than a single canonical algorithm. What unifies the pattern is the use of thresholds to create a more discriminative surrogate for downstream optimization.

## 2. Diffusion-based autonomous motion planning as an MDP

In "Non-differentiable Reward Optimization for Diffusion-based Autonomous Motion Planning" [2507.12977], the reverse diffusion process is cast as a Markov decision process in order to optimize safety and effectiveness objectives that are naturally non-differentiable. The training-time trajectory distribution is written as
$$
p_\theta(y^{0:K}) = p(y^K)\prod_{k=1}^K p_\theta(y^{k-1}\mid y^k,S_0,\hat F).
$$
A reward is placed on the final denoised trajectory $\hat y=y^0$ and extended across intermediate states, giving
$$
J(\theta)=\mathbb E_{p_\theta}\left[\sum_{k=0}^K R(y^k,\hat F)\right].
$$
By policy-gradient and importance sampling, the update direction becomes
$$
\nabla_\theta J=\mathbb E_{p_\theta}\left[\sum_{k=0}^K w_k(\theta,\theta_{\mathrm{old}})\cdot \nabla_\theta \log p_\theta(y^{k-1}\mid y^k,S_0,\hat F)\cdot A\right],
$$
with
$$
w_k(\theta,\theta_{\mathrm{old}})=\frac{p_\theta(y^{k-1}\mid \cdots)}{p_{\theta_{\mathrm{old}}}(y^{k-1}\mid \cdots)},\qquad
A=\frac{R-\bar R}{\sigma_R}.
$$

The paper motivates this construction by noting that diffusion planners are normally trained by maximizing a likelihood or reconstruction loss, whereas the key objectives in motion planning—collision avoidance and goal reaching—are expressed as non-differentiable metrics such as collision-rate and success-rate [2507.12977]. A pure policy-gradient treatment of reverse diffusion suffers from extremely sparse rewards in complex scenes, because sampled trajectories may rarely collide or rarely succeed initially. Reward-Weighted Dynamic Thresholding is introduced precisely to densify these objectives while keeping the optimization target aligned with the downstream planning criteria.

## 3. Per-metric reward shaping and dynamic threshold search

The core thresholding mechanism in the planning paper operates on multiple non-differentiable metrics. For metric $m$, the shaped per-sample reward is
$$
r_m^i(\epsilon_m)=
\begin{cases}
1 & \text{if } \mathrm{metric}_m(\mathrm{sample}_i)\ge \epsilon_m,\\
0 & \text{otherwise.}
\end{cases}
$$
These metric-wise rewards are combined as
$$
R=\sum_{m=1}^M \lambda_m\cdot \frac{1}{B}\sum_{i=1}^B r_m^i(\epsilon_m),
$$
with user-chosen importance weights $\lambda_m$ and batch size $B$. In the stepwise training procedure, the per-sample reward is written as
$$
R(i)=\sum_{m=1}^M \lambda_m\cdot r_m^i(\epsilon_m),
$$
followed by batch standardization through $\bar R$ and $\sigma_R$ [2507.12977].

Dynamic thresholding is used to prevent extreme sparsity. Let
$$
\mu_m(\epsilon)=\sum_{i=1}^B r_m^i(\epsilon).
$$
The aim is to find $\epsilon$ such that
$$
|\mu_m(\epsilon)-B/2|\le \delta.
$$
The update rule is
$$
\epsilon \leftarrow \epsilon + \mathrm{sign}\big((B/2)-\mu_m(\epsilon)\big)\cdot \alpha(j)\cdot \epsilon,
$$
where $j$ is the current iteration of the threshold search, $\alpha(j)$ is a decaying adaptation rate, and the number of sub-iterations is clamped to $J$ [2507.12977].

The paper attributes three effects to this design. First, it keeps the per-batch reward distribution away from the all-zero or all-one extremes, which raises the signal-to-noise ratio in the standardized advantage. Second, by targeting a 50% positive rate, each mini-batch contains both good and bad examples for the baseline, lowering gradient variance. Third, fixed thresholds are reported to be unstable because they can produce either vanishing gradients or no discrimination, whereas the dynamic scheme automatically finds an operating point that yields informative learning steps [2507.12977].

The reported empirical impact is specific. On CrowdNav, “dynamic thresholding + reward weighting” drove Collision Rate from $0.0628\to 0.0150$, Success Rate from $0.3540\to 0.4931$, and ADE from $0.2663\to 0.1984$, outperforming DTPP, DIPP, and Game-Theoretic baselines. On ETH-UCY, collision rate improved from approximately $0.0965\to 0.0879$ and success rate from $0.3635\to 0.3910$. Ablations are described as showing that per-metric thresholding, multi-reward aggregation, and policy-gradient weighting each contribute to stable learning and state-of-the-art safety/effectiveness trade-offs [2507.12977].

## 4. Conflict-aware thresholding in multi-reward policy optimization

In "GD$^2$PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization" [2606.16771], reward-weighted thresholding takes a different form. The setting is prompt-level group sampling under an old policy $\pi_{\bar\theta}$: for a query $q$, one samples rollouts $\tau=\{\tau_n\}_{n=1}^G$, each evaluated by $M$ separate reward functions $r_i(q,\tau_n)$. The reward-wise normalized advantages are
$$
A_n^i=
\frac{r_i(q,\tau_n)-\mathrm{mean}(r_i(q,\tau_1),\ldots,r_i(q,\tau_G))}
{\mathrm{std}(r_i(q,\tau_1),\ldots,r_i(q,\tau_G))+\epsilon_{\mathrm{adv}}},
$$
and the weighted aggregate advantage is
$$
A(\tau_n)\equiv A_n^{\mathrm{sum}}=\sum_{i=1}^M w_i A_n^i,
$$
with $w_i\ge 0$ [2606.16771].

The problem is multi-reward conflict: a single rollout may have positive advantages on some dimensions and negative advantages on others, so aggregation can cancel useful signal. GD$^2$PO therefore defines a rollout-level conflict metric $\Delta(\tau_n)$ and retains only rollouts satisfying a threshold test. Two variants are proposed:
$$
\Delta_{\mathrm{hard}}(\tau_n)=\mathbf 1_{\mathrm{sign}(A_n^i)=\mathrm{sign}(A_n^j)\ \mathrm{for\ all}\ i,j},
$$
and
$$
\mathrm{SNR}_n=
\frac{\left|\sum_{i=1}^M w_i A_n^i\right|}
{\left(\sum_{i=1}^M |w_i A_n^i|+\epsilon\right)},\qquad
\Delta_{\mathrm{SNR}}(\tau_n)=\mathrm{SNR}_n.
$$
Given threshold $\tau$, the mask is
$$
\delta(\tau_n)=
\begin{cases}
1 & \text{if } \Delta(\tau_n)\ge \tau,\\
0 & \text{otherwise.}
\end{cases}
$$
In the hard-filter variant, no extra threshold is used; in the SNR variant, $\delta_{\mathrm{SNR}}(\tau_n)=\mathbf 1_{\mathrm{SNR}_n>\tau}$ [2606.16771].

The method also introduces query-level reweighting. If
$$
\kappa(q)=\sum_{n=1}^G \delta(\tau_n),\qquad \hat \kappa(q)=\kappa(q)/G,
$$
then the final clipped surrogate objective is
$$
J(\theta)=
\mathbb E_{q,\{\tau_n\}}
\left[
\left(\frac{\hat\kappa(q)}{G}\right)
\sum_{n=1}^G
\left(\frac{1}{|\tau_n|}\right)
\sum_{t=1}^{|\tau_n|}
\gamma_n^t(\theta,\delta(\tau_n)\cdot A(\tau_n))
\right].
$$
The motivation given in the paper is that the variance of the per-query gradient estimate scales roughly as $1/\kappa(q)$ while its expected value scales as $\kappa(q)/G$, so $\hat\kappa(q)$ down-weights queries whose rollouts exhibit heavy multi-reward conflict [2606.16771].

The thresholding component is isolated by ablation. Varying $\tau$ in $\{0.2,0.5,0.8\}$ on a three-reward tool-calling task significantly outperforms vanilla GDPO for all tested values, with an intermediate $\tau\approx 0.5$ often yielding the best end-task accuracy. “Hard w/o QR” improves Correct-Call Acc by $+1.3\%$ over GDPO, and adding query reweighting (“Hard”) yields a further $+0.4\%$ gain. Training-curve comparisons show that GD$^2$PO-Hard and GD$^2$PO-SNR climb faster, in fewer gradient steps, to the same or higher final reward levels than GRPO and GDPO [2606.16771].

## 5. Threshold learning under censored feedback

"Learning Thresholds with Latent Values and Censored Feedback" [2312.04653] studies a formally different problem that nevertheless centers threshold selection under reward censoring. The learner chooses $\gamma\in[0,1]$, an unobserved latent value $v\in[0,1]$ is drawn from an unknown distribution $F$, and the only feedback is
$$
b(\gamma,v)=
\begin{cases}
g(\gamma,v) & \text{if } v\ge \gamma,\\
0 & \text{if } v<\gamma.
\end{cases}
$$
Equivalently, the learner observes $b=g(\gamma,v)\mathbf 1_{v\ge \gamma}$ and seeks
$$
\gamma^*\in \arg\max_{\gamma\in[0,1]} U(\gamma),
\qquad
U(\gamma)=\mathbb E_{v\sim F}[g(\gamma,v)\mathbf 1_{v\ge \gamma}].
$$

The paper considers three structural assumptions: monotonicity of $g$ in both $\gamma$ and $v$, an $L$-Lipschitz CDF for $F$, and one-sided Lipschitzness of $\gamma\mapsto g(\gamma,v)$ for each fixed $v$ [2312.04653]. Under mere monotonicity, the paper proves an impossibility result: for any fixed $\epsilon>0$, the number of queries required to find $\hat\gamma$ with $U(\hat\gamma)\ge U(\gamma^*)-\epsilon$ can be made arbitrarily large. The proof uses a “needle in a haystack” family in which the optimal threshold is localized at a latent point $\alpha$, but finitely many queries cannot reveal $\alpha$’s exact location [2312.04653].

Under stronger regularity, the picture changes sharply. The paper shows a tight query complexity $\tilde\Theta(1/\epsilon^3)$ when $g$ is monotone and the CDF of the value distribution is Lipschitz, and also when $g$ satisfies one-sided Lipschitzness. The active-learning template discretizes $[0,1]$ with grid step $\Delta=\epsilon/L$, estimates $U(\gamma)$ on the grid via repeated queries and Dvoretzky–Kiefer–Wolfowitz concentration, and returns the empirical maximizer. Matching lower bounds are derived using Hellinger-distance arguments [2312.04653].

The online extension yields a tight $\Theta(T^{2/3})$ regret bound. By discretizing into $K\approx T^{1/3}$ arms and applying an adversarial bandit method such as PolyINF, the algorithm obtains $O(T^{2/3})$ regret; an online-to-batch reduction gives the corresponding lower bound [2312.04653]. In this line of work, the threshold is not primarily a reward-shaping device. Rather, it is the action through which the learner interacts with censored rewards. This suggests a broader interpretation of thresholding: not only as a way to regularize gradients, but also as a way to parameterize the decision itself.

## 6. Global thresholding for visual generative alignment

"Threshold-Guided Optimization for Visual Generative Models" [2605.04653] develops a threshold-based surrogate for KL-regularized alignment when feedback is available as unpaired scalar scores rather than paired preferences. The starting objective is
$$
J(\pi)=\mathbb E_{x\sim D,\ y\sim \pi(\cdot|x)}[R(x,y)]-\beta\cdot \mathrm{KL}[\pi(\cdot|x)\Vert \pi_{\mathrm{ref}}(\cdot|x)].
$$
Its optimal policy satisfies
$$
\pi^*(y|x)\propto \pi_{\mathrm{ref}}(y|x)\cdot \exp(R(x,y)/\beta),
$$
with partition function
$$
Z(x)\coloneqq \int \pi_{\mathrm{ref}}(y'|x)\cdot \exp(R(x,y')/\beta)\,dy'.
$$
Taking logs gives
$$
\log \pi^*(y|x)-\log \pi_{\mathrm{ref}}(y|x)
=
\frac{1}{\beta}R(x,y)-\log Z(x)
\equiv \frac{1}{\beta}[R(x,y)-\tau^*(x)],
$$
where
$$
\tau^*(x)\coloneqq \beta\log Z(x).
$$
Hence
$$
\pi^*(y|x)>\pi_{\mathrm{ref}}(y|x)\iff R(x,y)>\tau^*(x).
$$
The difficulty is that $\tau^*(x)$ depends on an intractable integral [2605.04653].

Threshold-Guided Optimization replaces the instance-specific baseline by a single global threshold $\tau$. Given scalar scores $s_i\approx R(x_i,y_i)$, it defines pseudo-labels
$$
\ell_i=
\begin{cases}
1 & \text{if } s_i\ge \tau,\\
0 & \text{if } s_i<\tau,
\end{cases}
$$
and an implicit policy log-ratio
$$
\Delta_\theta(x,y)\coloneqq \beta[\log \pi_\theta(y|x)-\log \pi_{\mathrm{ref}}(y|x)].
$$
This yields the binary-classification loss
$$
L_{\mathrm{TGO}}(\theta;\tau)
=
\mathbb E_{(x,y,s)}
\left[
-\ell\log \sigma(\Delta_\theta)
-(1-\ell)\log(1-\sigma(\Delta_\theta))
\right].
$$
To improve sample efficiency, the paper introduces confidence weighting
$$
w(s;\tau)=1+c|s-\tau|,\qquad c\ge 0,
$$
so that the final surrogate becomes
$$
L_{\mathrm{TGO}}(\theta;\tau)
=
\mathbb E_{(x,y,s)}
\left[
-w(s;\tau)\cdot
\big(
\ell\log \sigma(\Delta_\theta)
+(1-\ell)\log(1-\sigma(\Delta_\theta))
\big)
\right].
$$
The threshold itself is set by a percentile $p$, for example the median:
$$
\tau \leftarrow \mathrm{Percentile}_p(\{s_i:(x_i,y_i)\in D\}),
$$
estimated once per epoch or even each batch [2605.04653].

The empirical setup spans diffusion models such as Stable Diffusion v1.5, MaskGIT-style token models such as Meissonic and FLUX, five reward models, and three text-to-image test sets, with extensions to text-to-video using Wan 1.3B and VideoReward [2605.04653]. The paper reports that TGO consistently outperforms SFT, CSFT, and pairwise baselines across all five reward models and three text-to-image benchmarks; score distributions exhibit a global right-shift; GPT-based human preference judgments favor TGO over Diffusion-KTO, DPO, and AlignProp; and the framework extends naturally to text-to-video [2605.04653].

## 7. Comparative interpretation and common misconceptions

A direct comparison of these works shows that the term “thresholding” covers materially different operations. In the motion-planning algorithm, the threshold is searched adaptively so that the batch-average count of positives is approximately $B/2$ [2507.12977]. In GD$^2$PO, the threshold is applied to a conflict metric, and in the SNR-based variant the threshold $\tau$ is chosen by small ablations such as $\tau\in\{0.2,0.5,0.8\}$; the paper explicitly notes that although $\tau$ is held fixed during training, the condition $\Delta(\tau_n)\ge \tau$ can be viewed as a per-batch adaptive gate because it depends on the current reward distributions [2606.16771]. In the latent-threshold problem, the threshold $\gamma$ is the learner’s action under censored feedback rather than an auxiliary training heuristic [2312.04653]. In TGO, the threshold is a data-driven approximation to the intractable oracle baseline $\tau^*(x)$ in a KL-regularized objective, and it is estimated from empirical score statistics [2605.04653].

These differences correct a common misconception: reward-weighted dynamic thresholding is not a universally standardized recipe. The phrase refers, in different papers, to adaptive reward densification, conflict-aware masking, active threshold learning, or threshold-guided surrogate construction. Another misconception is that thresholding always exists to handle sparsity. In the motion-planning setting, sparsity is central; in GD$^2$PO, the emphasis is on preventing cancellation from multi-reward disagreement; in the censored-feedback problem, the challenge is partial observability of rewards; and in TGO, the threshold compensates for the absence of paired comparisons.

A plausible unifying view is that all four formulations use thresholds to create a controllable interface between raw feedback and optimization. The control variable differs—positive-rate balance, keep ratio, censored utility, or score quantile—but the methodological role is similar: thresholding determines which samples, rewards, or actions are informative enough to shape the update.

Source: https://www.emergentmind.com/topics/reward-weighted-dynamic-thresholding