SiMPO: Signed Measure Policy Optimization in RL

• SiMPO is a reinforcement learning framework that reformulates policy improvement as a two-stage measure matching problem, allowing the use of signed target measures.
• It mitigates over-greedy updates by incorporating negative weights to actively repel low-Q actions, thus addressing key limitations of advantage-weighted regression.
• By employing reweighted diffusion or flow matching, SiMPO demonstrates improved performance in diverse tasks such as MuJoCo locomotion and DNA sequence generation.

Signed Measure Policy Optimization (SiMPO) is a framework for online reinforcement learning with diffusion and flow policies that reformulates policy improvement as two-stage measure matching rather than as conventional softmax-based advantage reweighting. It is introduced in “SiMPO: Measure Matching for Online Diffusion Reinforcement Learning” (Ma et al., 10 Mar 2026). The central claim is that widely used “advantage-weighted regression” style methods in diffusion RL inherit two persistent pathologies: softmax reweighting over the behavior policy produces overly greedy updates that collapse onto a few high-advantage samples, and negative feedback from low-reward samples is essentially ignored. SiMPO addresses both issues by first constructing a virtual target policy through $f$-divergence regularized policy optimization and then fitting diffusion or flow models to that target via reweighted matching, while explicitly allowing the target to be a normalized signed measure. This permits negative weights, so low-$Q$ actions can contribute repulsive feedback rather than merely receiving negligible mass (Ma et al., 10 Mar 2026).

1. Problem setting and motivation

SiMPO is situated in the online diffusion RL setting in which a behavior policy $\pi_{\rm old}$ is modeled by a diffusion or flow network and iteratively improved using data collected online. At each iteration, the agent rolls out $\pi_{\rm old}$ in the environment, obtains state-action pairs and rewards or $Q$-values, constructs a target update that reweights these samples by their desirability, and retrains the diffusion or flow predictor to match this target (Ma et al., 10 Mar 2026).

The paper identifies a specific failure mode in the dominant KL-regularized mirror-descent lineage. In that formulation, the optimal reweighting of the current policy is proportional to $\exp(Q/\lambda)$, and diffusion RL variants of advantage-weighted regression adopt this forward-KL solution by reweighting the diffusion loss with $\operatorname{softmax}(\text{advantage})$. According to the paper, exponential weights concentrate mass onto a few high-advantage samples, making updates over-greedy and brittle, while assigning negligible weight to the large set of negative samples. The stated consequences are that useful repulsive feedback is ignored, learning can be trapped in local optima, and exploration is impaired (Ma et al., 10 Mar 2026).

A key conceptual shift in SiMPO is that policy improvement is treated as a measure-construction problem followed by a projection problem. This suggests that the usual insistence on nonnegative target densities at the intermediate stage may be stronger than necessary for training diffusion or flow models. In the SiMPO formulation, what must be preserved is normalization of the target measure, whereas positivity can be relaxed in Stage I.

2. Two-stage measure matching formulation

The first stage constructs a virtual target policy by solving an $f$-divergence regularized policy optimization problem. For a convex generator $f:\mathbb{R}_+\to\mathbb{R}$ with $f(1)=0$, the $Q$0-divergence between distributions $Q$1 and $Q$2 is

$Q$3

At a state $Q$4, the regularized optimization problem is

$Q$5

and, under mild conditions, the Lagrangian yields the closed-form target

$Q$6

Here $Q$7 enforces normalization and $Q$8 controls proximity to $Q$9 (Ma et al., 10 Mar 2026).

The forward-KL case recovers the standard softmax update. With $\pi_{\rm old}$0, $\pi_{\rm old}$1 and $\pi_{\rm old}$2, so

$\pi_{\rm old}$3

SiMPO then relaxes the non-negativity constraint in Stage I. The paper states that it suffices for the target to be a normalized signed measure, and that the definition of $\pi_{\rm old}$4-divergence extends to signed measures $\pi_{\rm old}$5 against a probability measure $\pi_{\rm old}$6 when $\pi_{\rm old}$7 and $\pi_{\rm old}$8 is measurable on $\pi_{\rm old}$9. Motivated by the observation that the reweighting function $\pi_{\rm old}$0 is the essential design degree of freedom, SiMPO directly generalizes to monotone increasing weighting functions and permits negative values:

$\pi_{\rm old}$1

The only constraint at this stage is $\pi_{\rm old}$2; positivity is not required. The resulting signed target measure can therefore assign negative weights to low-$\pi_{\rm old}$3 actions, encoding repulsive feedback (Ma et al., 10 Mar 2026).

3. Reweighted diffusion and flow matching

The second stage projects the virtual target back into the model class by reweighted conditional matching. The training loss is

$\pi_{\rm old}$4

with weights $\pi_{\rm old}$5. Here $\pi_{\rm old}$6, $\pi_{\rm old}$7 is a forward noising bridge for diffusion or flow, $\pi_{\rm old}$8 is Gaussian noise, and $\pi_{\rm old}$9 predicts the velocity or score field (Ma et al., 10 Mar 2026).

The paper gives a closed-form optimum for the reweighted conditional loss:

$Q$0

Negative weights enter linearly. In the interpretation advanced by the paper, this means that low-value samples do not merely fail to attract the predictor; instead, their conditional velocity contributions change sign and actively repel the fit away from those actions.

SiMPO allows arbitrary monotone increasing weighting functions of the form $Q$1. The paper lists several instances:

Scheme	Weighting function	Note
Softmax (forward KL)	$Q$2	Recovers AWR-style reweighting
$Q$3-divergence (QVPO-like)	$Q$4	QVPO uses weights $Q$5
$Q$6-divergences ($Q$7)	$Q$8	Produces power-law weights
Negative-aware design wd1	$Q$9	Monotonic signed weighting

The normalization term $\exp(Q/\lambda)$0 is estimated from a group of sampled actions per state. For exponential weights this is softmax normalization; for power weights the procedure computes the active set size, defined as the non-clipped samples, and solves for $\exp(Q/\lambda)$1 so that $\exp(Q/\lambda)$2. For signed weights, the paper uses truncation to stabilize training:

$\exp(Q/\lambda)$3

It also states that the total measure is kept positive (Ma et al., 10 Mar 2026).

4. Signed measures, repulsion, and geometric interpretation

A central distinctive feature of SiMPO is the use of signed measures. In the paper’s geometric treatment, the local reweighted objective at fixed $\exp(Q/\lambda)$4 is

$\exp(Q/\lambda)$5

which can be rearranged as

$\exp(Q/\lambda)$6

When $\exp(Q/\lambda)$7, the minimizer is the weighted average $\exp(Q/\lambda)$8, corresponding to the closed-form solution above. In that regime, negative weights subtract their velocity components from the average, pushing the fitted velocity away from directions associated with low-$\exp(Q/\lambda)$9 actions. The paper interprets this as a direct repulsion mechanism (Ma et al., 10 Mar 2026).

When $\operatorname{softmax}(\text{advantage})$0 at a location, the paper states that the objective becomes unbounded from below and gradients repel the network away from matching those directions, producing a “repulsive barrier” around negatively weighted regions. This point is important because it clarifies that negative weighting is not merely a signed analogue of importance sampling; it changes the geometry of the regression objective itself.

The same section gives a probability-path interpretation. For $\operatorname{softmax}(\text{advantage})$1, the induced velocity field transports the density along a valid reweighted probability path,

$\operatorname{softmax}(\text{advantage})$2

so negative mass in the integrand must be offset by larger positive mass elsewhere. The paper states that this amplifies probabilities on high-$\operatorname{softmax}(\text{advantage})$3 actions, sharpening attraction to good actions while actively repelling from bad ones. A common misconception is therefore that signed reweighting simply “breaks” the probabilistic interpretation. The paper’s position is more specific: Stage I uses a normalized signed target measure, while Stage II yields a valid reweighted probability path when the relevant denominator remains positive.

5. Theoretical guarantees and relation to prior methods

SiMPO provides a policy-improvement guarantee under signed-measure normalization. If $\operatorname{softmax}(\text{advantage})$4 is strictly increasing and the target measure is normalized, then for every state $\operatorname{softmax}(\text{advantage})$5,

$\operatorname{softmax}(\text{advantage})$6

with the proof proceeding through covariance positivity, $\operatorname{softmax}(\text{advantage})$7, where $\operatorname{softmax}(\text{advantage})$8 (Ma et al., 10 Mar 2026). This result is used to justify the claim that monotone signed reweighting remains improvement-oriented even though the intermediate target is not required to be nonnegative.

The framework is explicitly presented as a generalization of several existing methods. AWR and DPMD correspond to the forward-KL case with exponential weighting. SiMPO-Exp retains the same $\operatorname{softmax}(\text{advantage})$9 weighting but places it inside the signed-measure and two-stage matching framework. QVPO appears as an $f$0-divergence special case: with $f$1, SiMPO reproduces a clipped linear weighting $f$2, explaining QVPO’s behavior as a case in which negative samples are zeroed out rather than given repulsive influence. The paper also derives power-law weights from $f$3-divergences, with tuning of $f$4 controlling curvature and interpolating between flatter and steeper weighting schemes.

The comparison to negative-aware preference optimization is also explicit. Methods such as wd1 are described as introducing signed weights ad hoc, whereas SiMPO presents them as instances of monotone signed measure matching with a geometric interpretation and a policy-improvement guarantee under normalization. By contrast, reverse-process RL methods for diffusion based on policy gradients or backpropagation through sampling are described as computationally intensive and “mode-seeking” due to reverse-KL. SiMPO preserves the standard forward process and training infrastructure, which the paper positions as a more efficient alternative (Ma et al., 10 Mar 2026).

6. Empirical behavior, implementation details, and limitations

The empirical evaluation spans bandits, MuJoCo v4 locomotion, and DNA sequence generation (Ma et al., 10 Mar 2026). In bandits, the metric is expected regret and the focus is exploration and reward landscape curvature. In MuJoCo v4 locomotion, the tasks are HalfCheetah, Humanoid, Ant, Walker2d, Hopper, and Swimmer, with average return over episodes, $f$5M environment steps, and five seeds. In DNA sequence generation, the setting is discrete diffusion for enhancer design, measured with Pred-Activity and compared to controlled sampling methods CG, SMC, TDS, and CFG, as well as RL baselines DRAKES and RL-D².

Several quantitative findings are stated directly. In the bandit exploration study, linear, square, and exponential weighting without negative weights often get stuck in local optima; adding negative truncation, reported as Lin. Neg. with $f$6, escapes sub-optima and reduces regret. In the reward-landscape study, square reweighting, described as an $f$7-divergence with $f$8, yields the lowest, stable regret for broad, flat optima, while linear reweighting performs best for sharp optima; exponential weighting is typically inferior. In MuJoCo, SiMPO variants consistently outperform diffusion RL baselines QVPO, DIPO, DACER, and QSM and are competitive with SAC and TD3, with SiMPO-Linear and SiMPO-Lin. Neg. achieving top or tied-top scores on HalfCheetah and Humanoid. In DNA generation, SiMPO-Lin. Neg. and SiMPO-Sqr. Neg. reach $f$9 and $f:\mathbb{R}_+\to\mathbb{R}$0 Pred-Activity, reported as $f:\mathbb{R}_+\to\mathbb{R}$1 to $f:\mathbb{R}_+\to\mathbb{R}$2 over the best baseline, RL-D² at $f:\mathbb{R}_+\to\mathbb{R}$3. The paper further states that standard SiMPO variants, Linear and Square, already match or surpass RL-D², and that the largest boost comes from negative reweighting (Ma et al., 10 Mar 2026).

The algorithmic procedure is correspondingly simple. For each state $f:\mathbb{R}_+\to\mathbb{R}$4, one samples $f:\mathbb{R}_+\to\mathbb{R}$5 actions $f:\mathbb{R}_+\to\mathbb{R}$6, evaluates $f:\mathbb{R}_+\to\mathbb{R}$7, computes $f:\mathbb{R}_+\to\mathbb{R}$8 to satisfy normalization, chooses temperature $f:\mathbb{R}_+\to\mathbb{R}$9, computes weights $f(1)=0$0, optionally applies negative truncation, and then trains $f(1)=0$1 with the weighted matching loss. After that, the updated policy is rolled out, critics or $f(1)=0$2-functions are updated, and the process repeats.

The paper also lists several stability mechanisms. Negative weights are truncated with a lower bound $f(1)=0$3. Group-based normalization is used for $f(1)=0$4, including handling of negative values. Temperature can be autotuned through the dual objective

$f(1)=0$5

whose gradient equals the entropy of the softmax$f(1)=0$6. The reported ablations show robustness for KL constraints approximately $f(1)=0$7 to $f(1)=0$8, with KL approximately $f(1)=0$9 working well across tasks (Ma et al., 10 Mar 2026).

The limitations are stated with similar specificity. The temperature $Q$00 or KL constraint and the normalization term $Q$01 matter, although autotuning and group-based normalization help. Signed weights require care: truncation, ensuring positive total measure, and monitoring $Q$02 are important. Power-law weights are described as less brittle than exponential weights in noisy settings, while excessive negativity may destabilize training. Some environments, including Swimmer, show limited gains. Negative reweighting improves some tasks, notably HalfCheetah and Humanoid, but not all; the paper therefore recommends tuning $Q$03 to task-specific reward curvature. It identifies learned weighting functions $Q$04, state-dependent $Q$05, combination with exploration bonuses, and deeper convergence theory with negative weights as future directions. This suggests that SiMPO is best understood not as a single fixed algorithmic instantiation, but as a general policy-improvement template for diffusion and flow RL grounded in signed measure construction and reweighted matching.