Reward-Weighted Dynamic Thresholding

• The paper demonstrates adaptive thresholding techniques that densify sparse and non-differentiable rewards, enabling smoother gradient estimation in complex tasks.
• It outlines methodologies including metric-specific thresholds, conflict-aware masking, and active threshold learning to enhance training signals across diffusion planning, multi-reward optimization, and visual generative alignment.
• Collectively, these approaches offer actionable insights for optimizing safety, effectiveness, and alignment in various machine learning and control applications.

Searching arXiv for the cited work and closely related thresholding/reward-optimization papers. Reward-Weighted Dynamic Thresholding denotes a cluster of threshold-centered optimization procedures in which rewards, advantages, or scalar feedback are transformed before they are used for learning. In recent arXiv work, the phrase is used most explicitly for diffusion-based autonomous motion planning, where adaptively tuned thresholds convert non-differentiable safety and effectiveness metrics into shaped rewards that weight diffusion log-likelihood gradients (Lee et al., 17 Jul 2025). Closely related thresholding mechanisms also appear in multi-reward policy optimization for LLM post-training, where rollout-level masks are applied to conflict-prone advantage vectors (Liu et al., 15 Jun 2026), in latent-threshold learning with censored feedback, where the threshold itself is the learned decision variable (Zhang et al., 2023), and in visual generative alignment, where a global threshold replaces an intractable instance-specific KL baseline (Bai et al., 6 May 2026). These formulations are not identical, but they share a common concern: direct optimization is hindered by sparsity, censoring, or interference, and thresholding is used to recover an informative training signal.

1. Scope and recurring structure

Across the cited literature, thresholding enters the learning problem in several distinct ways.

Setting	Thresholded quantity	Function
Diffusion motion planning	Per-metric threshold $\epsilon_m$	Shapes dense rewards and balances positives near $B/2$
GD$^2$PO	Conflict threshold $\tau$	Masks rollouts with severe reward-wise disagreement
Latent-value learning	Decision threshold $\gamma$	Maximizes censored utility $U(\gamma)$
TGO	Global score threshold $\tau$	Forms pseudo-labels from unpaired scalar scores

In the motion-planning formulation, the threshold is explicitly adaptive inside training: each $\epsilon_m$ is updated so that the number of positives for metric $m$ is approximately half the batch, thereby avoiding all-zero or all-one rewards (Lee et al., 17 Jul 2025). In GD$^2$PO, the threshold is applied to a rollout conflict metric, either a hard sign-consistency indicator or an SNR-based coherence score, in order to suppress advantage cancellation across reward dimensions (Liu et al., 15 Jun 2026). In the latent-threshold problem, the learner repeatedly queries a threshold and receives censored rewards of the form $B/2$0, so the threshold is the central object of estimation (Zhang et al., 2023). In Threshold-Guided Optimization, a global percentile-based threshold turns scalar reward scores into binary pseudo-labels and can be recomputed once per epoch or per batch (Bai et al., 6 May 2026).

A plausible implication is that “reward-weighted dynamic thresholding” is best treated as a methodological pattern rather than a single canonical algorithm. What unifies the pattern is the use of thresholds to create a more discriminative surrogate for downstream optimization.

2. Diffusion-based autonomous motion planning as an MDP

In "Non-differentiable Reward Optimization for Diffusion-based Autonomous Motion Planning" (Lee et al., 17 Jul 2025), the reverse diffusion process is cast as a Markov decision process in order to optimize safety and effectiveness objectives that are naturally non-differentiable. The training-time trajectory distribution is written as

$B/2$1

A reward is placed on the final denoised trajectory $B/2$2 and extended across intermediate states, giving

$B/2$3

By policy-gradient and importance sampling, the update direction becomes

$B/2$4

with

$B/2$5

The paper motivates this construction by noting that diffusion planners are normally trained by maximizing a likelihood or reconstruction loss, whereas the key objectives in motion planning—collision avoidance and goal reaching—are expressed as non-differentiable metrics such as collision-rate and success-rate (Lee et al., 17 Jul 2025). A pure policy-gradient treatment of reverse diffusion suffers from extremely sparse rewards in complex scenes, because sampled trajectories may rarely collide or rarely succeed initially. Reward-Weighted Dynamic Thresholding is introduced precisely to densify these objectives while keeping the optimization target aligned with the downstream planning criteria.

3. Per-metric reward shaping and dynamic threshold search

The core thresholding mechanism in the planning paper operates on multiple non-differentiable metrics. For metric $B/2$6, the shaped per-sample reward is

$B/2$7

These metric-wise rewards are combined as

$B/2$8

with user-chosen importance weights $B/2$9 and batch size $^2$0. In the stepwise training procedure, the per-sample reward is written as

$^2$1

followed by batch standardization through $^2$2 and $^2$3 (Lee et al., 17 Jul 2025).

Dynamic thresholding is used to prevent extreme sparsity. Let

$^2$4

The aim is to find $^2$5 such that

$^2$6

The update rule is

$^2$7

where $^2$8 is the current iteration of the threshold search, $^2$9 is a decaying adaptation rate, and the number of sub-iterations is clamped to $\tau$0 (Lee et al., 17 Jul 2025).

The paper attributes three effects to this design. First, it keeps the per-batch reward distribution away from the all-zero or all-one extremes, which raises the signal-to-noise ratio in the standardized advantage. Second, by targeting a 50% positive rate, each mini-batch contains both good and bad examples for the baseline, lowering gradient variance. Third, fixed thresholds are reported to be unstable because they can produce either vanishing gradients or no discrimination, whereas the dynamic scheme automatically finds an operating point that yields informative learning steps (Lee et al., 17 Jul 2025).

The reported empirical impact is specific. On CrowdNav, “dynamic thresholding + reward weighting” drove Collision Rate from $\tau$1, Success Rate from $\tau$2, and ADE from $\tau$3, outperforming DTPP, DIPP, and Game-Theoretic baselines. On ETH-UCY, collision rate improved from approximately $\tau$4 and success rate from $\tau$5. Ablations are described as showing that per-metric thresholding, multi-reward aggregation, and policy-gradient weighting each contribute to stable learning and state-of-the-art safety/effectiveness trade-offs (Lee et al., 17 Jul 2025).

4. Conflict-aware thresholding in multi-reward policy optimization

In "GD$\tau$6PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization" (Liu et al., 15 Jun 2026), reward-weighted thresholding takes a different form. The setting is prompt-level group sampling under an old policy $\tau$7: for a query $\tau$8, one samples rollouts $\tau$9, each evaluated by $\gamma$0 separate reward functions $\gamma$1. The reward-wise normalized advantages are

$\gamma$2

and the weighted aggregate advantage is

$\gamma$3

with $\gamma$4 (Liu et al., 15 Jun 2026).

The problem is multi-reward conflict: a single rollout may have positive advantages on some dimensions and negative advantages on others, so aggregation can cancel useful signal. GD$\gamma$5PO therefore defines a rollout-level conflict metric $\gamma$6 and retains only rollouts satisfying a threshold test. Two variants are proposed:

$\gamma$7

and

$\gamma$8

Given threshold $\gamma$9, the mask is

$U(\gamma)$0

In the hard-filter variant, no extra threshold is used; in the SNR variant, $U(\gamma)$1 (Liu et al., 15 Jun 2026).

The method also introduces query-level reweighting. If

$U(\gamma)$2

then the final clipped surrogate objective is

$U(\gamma)$3

The motivation given in the paper is that the variance of the per-query gradient estimate scales roughly as $U(\gamma)$4 while its expected value scales as $U(\gamma)$5, so $U(\gamma)$6 down-weights queries whose rollouts exhibit heavy multi-reward conflict (Liu et al., 15 Jun 2026).

The thresholding component is isolated by ablation. Varying $U(\gamma)$7 in $U(\gamma)$8 on a three-reward tool-calling task significantly outperforms vanilla GDPO for all tested values, with an intermediate $U(\gamma)$9 often yielding the best end-task accuracy. “Hard w/o QR” improves Correct-Call Acc by $\tau$0 over GDPO, and adding query reweighting (“Hard”) yields a further $\tau$1 gain. Training-curve comparisons show that GD$\tau$2PO-Hard and GD$\tau$3PO-SNR climb faster, in fewer gradient steps, to the same or higher final reward levels than GRPO and GDPO (Liu et al., 15 Jun 2026).

5. Threshold learning under censored feedback

"Learning Thresholds with Latent Values and Censored Feedback" (Zhang et al., 2023) studies a formally different problem that nevertheless centers threshold selection under reward censoring. The learner chooses $\tau$4, an unobserved latent value $\tau$5 is drawn from an unknown distribution $\tau$6, and the only feedback is

$\tau$7

Equivalently, the learner observes $\tau$8 and seeks

$\tau$9

The paper considers three structural assumptions: monotonicity of $\epsilon_m$0 in both $\epsilon_m$1 and $\epsilon_m$2, an $\epsilon_m$3-Lipschitz CDF for $\epsilon_m$4, and one-sided Lipschitzness of $\epsilon_m$5 for each fixed $\epsilon_m$6 (Zhang et al., 2023). Under mere monotonicity, the paper proves an impossibility result: for any fixed $\epsilon_m$7, the number of queries required to find $\epsilon_m$8 with $\epsilon_m$9 can be made arbitrarily large. The proof uses a “needle in a haystack” family in which the optimal threshold is localized at a latent point $m$0, but finitely many queries cannot reveal $m$1’s exact location (Zhang et al., 2023).

Under stronger regularity, the picture changes sharply. The paper shows a tight query complexity $m$2 when $m$3 is monotone and the CDF of the value distribution is Lipschitz, and also when $m$4 satisfies one-sided Lipschitzness. The active-learning template discretizes $m$5 with grid step $m$6, estimates $m$7 on the grid via repeated queries and Dvoretzky–Kiefer–Wolfowitz concentration, and returns the empirical maximizer. Matching lower bounds are derived using Hellinger-distance arguments (Zhang et al., 2023).

The online extension yields a tight $m$8 regret bound. By discretizing into $m$9 arms and applying an adversarial bandit method such as PolyINF, the algorithm obtains $^2$0 regret; an online-to-batch reduction gives the corresponding lower bound (Zhang et al., 2023). In this line of work, the threshold is not primarily a reward-shaping device. Rather, it is the action through which the learner interacts with censored rewards. This suggests a broader interpretation of thresholding: not only as a way to regularize gradients, but also as a way to parameterize the decision itself.

6. Global thresholding for visual generative alignment

"Threshold-Guided Optimization for Visual Generative Models" (Bai et al., 6 May 2026) develops a threshold-based surrogate for KL-regularized alignment when feedback is available as unpaired scalar scores rather than paired preferences. The starting objective is

$^2$1

Its optimal policy satisfies

$^2$2

with partition function

$^2$3

Taking logs gives

$^2$4

where

$^2$5

Hence

$^2$6

The difficulty is that $^2$7 depends on an intractable integral (Bai et al., 6 May 2026).

Threshold-Guided Optimization replaces the instance-specific baseline by a single global threshold $^2$8. Given scalar scores $^2$9, it defines pseudo-labels

$B/2$00

and an implicit policy log-ratio

$B/2$01

This yields the binary-classification loss

$B/2$02

To improve sample efficiency, the paper introduces confidence weighting

$B/2$03

so that the final surrogate becomes

$B/2$04

The threshold itself is set by a percentile $B/2$05, for example the median:

$B/2$06

estimated once per epoch or even each batch (Bai et al., 6 May 2026).

The empirical setup spans diffusion models such as Stable Diffusion v1.5, MaskGIT-style token models such as Meissonic and FLUX, five reward models, and three text-to-image test sets, with extensions to text-to-video using Wan 1.3B and VideoReward (Bai et al., 6 May 2026). The paper reports that TGO consistently outperforms SFT, CSFT, and pairwise baselines across all five reward models and three text-to-image benchmarks; score distributions exhibit a global right-shift; GPT-based human preference judgments favor TGO over Diffusion-KTO, DPO, and AlignProp; and the framework extends naturally to text-to-video (Bai et al., 6 May 2026).

7. Comparative interpretation and common misconceptions

A direct comparison of these works shows that the term “thresholding” covers materially different operations. In the motion-planning algorithm, the threshold is searched adaptively so that the batch-average count of positives is approximately $B/2$07 (Lee et al., 17 Jul 2025). In GD$B/2$08PO, the threshold is applied to a conflict metric, and in the SNR-based variant the threshold $B/2$09 is chosen by small ablations such as $B/2$10; the paper explicitly notes that although $B/2$11 is held fixed during training, the condition $B/2$12 can be viewed as a per-batch adaptive gate because it depends on the current reward distributions (Liu et al., 15 Jun 2026). In the latent-threshold problem, the threshold $B/2$13 is the learner’s action under censored feedback rather than an auxiliary training heuristic (Zhang et al., 2023). In TGO, the threshold is a data-driven approximation to the intractable oracle baseline $B/2$14 in a KL-regularized objective, and it is estimated from empirical score statistics (Bai et al., 6 May 2026).

These differences correct a common misconception: reward-weighted dynamic thresholding is not a universally standardized recipe. The phrase refers, in different papers, to adaptive reward densification, conflict-aware masking, active threshold learning, or threshold-guided surrogate construction. Another misconception is that thresholding always exists to handle sparsity. In the motion-planning setting, sparsity is central; in GD$B/2$15PO, the emphasis is on preventing cancellation from multi-reward disagreement; in the censored-feedback problem, the challenge is partial observability of rewards; and in TGO, the threshold compensates for the absence of paired comparisons.

A plausible unifying view is that all four formulations use thresholds to create a controllable interface between raw feedback and optimization. The control variable differs—positive-rate balance, keep ratio, censored utility, or score quantile—but the methodological role is similar: thresholding determines which samples, rewards, or actions are informative enough to shape the update.