Repulsive Reparameterization Overview

• Repulsive reparameterization is a family of variable transformations that enforce constraints and prevent undesirable behaviors such as boundary violations and mode collapse.
• It is applied in domains like constrained optimization, generative sampling, and discrete latent-variable learning to preserve feasibility and improve diversity.
• By leveraging geometric properties and shared randomness, these methods enhance stability, reduce variance, and enable exact constraint satisfaction in complex models.

Searching arXiv for recent and directly relevant papers on “repulsive reparameterization” and neighboring formulations. Repulsive reparameterization denotes a class of constructions in which a change of variables, output ansatz, or latent-coordinate transformation makes undesirable behavior difficult to realize in the original coordinates. Depending on the field, the effect may be boundary avoidance through vanishing Jacobians, exact constraint satisfaction by construction, latent proposals that preserve diversity and raise acceptance, nuisance–target decorrelation, or low-variance coupling through shared randomness. The phrase does not identify a single canonical algorithm across the literature. Rather, the cited work shows several technically distinct uses of reparameterization that produce a “repulsive” effect only in specific senses: geometric and kinematic in constrained optimization, diversity-preserving in generative sampling, variance-reducing in discrete latent-variable learning, and stabilizing in PINNs, Bayesian inference, reinforcement learning, and rendering (Leplat, 26 Aug 2025, Wang et al., 2021, Tokui et al., 2016, Nand et al., 2023, Paradiso et al., 2024, Wu et al., 13 May 2025, Zhong et al., 8 Aug 2025).

1. Terminological scope and recurring mechanisms

Across current usage, repulsive reparameterization is best treated as a family of related mechanisms rather than a standardized term. In the strongest sense, a reparameterization is “repulsive” when it makes the boundary effectively non-penetrable without adding an explicit barrier term. In weaker senses, it may preserve mode coverage, reduce collapse into concentrated regions, or suppress nuisance or variance transfer between coupled components.

Domain	Reparameterized object	Effect
Constrained optimization	Feasible set $x=g(u)$	Vanishing Jacobian, null speed
GAN sampling	Markov chain in latent space	Dependent proposals, higher acceptance
Discrete latent-variable learning	Discrete variable marginalized under shared noise	Common-random-numbers variance reduction
PINNs and Bayesian inference	Solution ansatz or nuisance coordinates	Exact BCs or decorrelation
RL and rendering	Policy action or BRDF integral	Stable reuse or support-preserving transport

A common misconception is that repulsive reparameterization must introduce an explicit repulsive force or diversity regularizer. The constrained-optimization formulation shows the opposite: the boundary effect can arise purely from geometry, through maps whose Jacobians vanish on active faces or lose rank on the simplex boundary, so that primal motion in forbidden directions disappears kinematically (Leplat, 26 Aug 2025). Conversely, several other papers explicitly note that their methods are not “repulsive” in the determinantal or barrier sense, even though they improve diversity, support coverage, or stability (Wang et al., 2021, Tokui et al., 2016, Wu et al., 13 May 2025).

2. Geometry-induced repulsion in constrained optimization

The most direct formulation appears in "The Geometry of Constrained Optimization: Constrained Gradient Flows via Reparameterization: A-Stable Implicit Schemes, KKT from Stationarity, and Geometry-Respecting Algorithms" (Leplat, 26 Aug 2025). There, the central construction is a feasible-set map $x=g(u)$ and the induced flow

$$\dot u(t) = -\nabla_x \Phi(g(u(t))), \qquad \dot x(t)=J_g(u(t))\,\dot u(t)= -J_g(u(t))\,\nabla_x\Phi(x(t)).$$

The paper emphasizes that feasibility is enforced by smooth maps $x=g(u)$ or by intrinsic manifold geometry, so the optimization trajectory never leaves the feasible set.

For the orthant $\mathbb{R}_+^n$, the paper uses componentwise positive maps

$$g(u)=\exp(u) \quad\text{or}\quad g(u)=\log\cosh(u),$$

with exponential map Jacobian

$$J_g(u)=\operatorname{diag}(e^{u_i})=\operatorname{diag}(x_i).$$

The primal flow becomes

$\dot x = -\operatorname{diag}(x)\,\nabla \Phi(x),$

so if $x_i\to 0$, then $\dot x_i\to 0$. This is the paper’s kinematic slackness identity,

$x=g(u)$0

and at stationarity it yields

$x=g(u)$1

The repulsive effect here is not a barrier potential; it is null speed at the boundary.

For the box $x=g(u)$2, the map is

$x=g(u)$3

with

$x=g(u)$4

The induced flow

$x=g(u)$5

has a Jacobian that smoothly vanishes at the lower and upper faces, damping the normal component of motion as an active bound is approached.

For the simplex $x=g(u)$6, the softmax map

$x=g(u)$7

has Jacobian

$x=g(u)$8

which is PSD, has rank $x=g(u)$9, and satisfies $$\dot u(t) = -\nabla_x \Phi(g(u(t))), \qquad \dot x(t)=J_g(u(t))\,\dot u(t)= -J_g(u(t))\,\nabla_x\Phi(x(t)).$$0. The induced dynamics are the replicator or Fisher-natural-gradient flow

$$\dot u(t) = -\nabla_x \Phi(g(u(t))), \qquad \dot x(t)=J_g(u(t))\,\dot u(t)= -J_g(u(t))\,\nabla_x\Phi(x(t)).$$1

Thus the reparameterization embeds the tangent-space projection into the dynamics themselves.

The paper extends the same geometry-respecting viewpoint to the Stiefel manifold $$\dot u(t) = -\nabla_x \Phi(g(u(t))), \qquad \dot x(t)=J_g(u(t))\,\dot u(t)= -J_g(u(t))\,\nabla_x\Phi(x(t)).$$2, where feasibility is preserved intrinsically rather than by a Euclidean $$\dot u(t) = -\nabla_x \Phi(g(u(t))), \qquad \dot x(t)=J_g(u(t))\,\dot u(t)= -J_g(u(t))\,\nabla_x\Phi(x(t)).$$3. Its gradient is

$$\dot u(t) = -\nabla_x \Phi(g(u(t))), \qquad \dot x(t)=J_g(u(t))\,\dot u(t)= -J_g(u(t))\,\nabla_x\Phi(x(t)).$$4

or equivalently

$$\dot u(t) = -\nabla_x \Phi(g(u(t))), \qquad \dot x(t)=J_g(u(t))\,\dot u(t)= -J_g(u(t))\,\nabla_x\Phi(x(t)).$$5

Stationarity coincides with KKT across these constraint classes. The paper states that complementary slackness emerges kinematically, either through “null speed” or through “constant dual speed with vanishing Jacobian.” It then discretizes the flows with backward Euler or trapezoidal/Cayley schemes, solved by Modified Gauss–Newton or KL-prox inner loops, and reports numerical tests on NNLS, simplex- and box-constrained least squares, and orthogonality demonstrating stability, accuracy, and runtime efficiency (Leplat, 26 Aug 2025).

3. Latent-space dependent proposals and diversity-preserving sampling

In generative modeling, the closest analogue is "Reparameterized Sampling for Generative Adversarial Networks" (Wang et al., 2021). REP-GAN reparameterizes the Markov chain into the generator’s latent space rather than proposing directly in sample space. If $$\dot u(t) = -\nabla_x \Phi(g(u(t))), \qquad \dot x(t)=J_g(u(t))\,\dot u(t)= -J_g(u(t))\,\nabla_x\Phi(x(t)).$$6 with $$\dot u(t) = -\nabla_x \Phi(g(u(t))), \qquad \dot x(t)=J_g(u(t))\,\dot u(t)= -J_g(u(t))\,\nabla_x\Phi(x(t)).$$7, the coupled chain proposes $$\dot u(t) = -\nabla_x \Phi(g(u(t))), \qquad \dot x(t)=J_g(u(t))\,\dot u(t)= -J_g(u(t))\,\nabla_x\Phi(x(t)).$$8, maps $$\dot u(t) = -\nabla_x \Phi(g(u(t))), \qquad \dot x(t)=J_g(u(t))\,\dot u(t)= -J_g(u(t))\,\nabla_x\Phi(x(t)).$$9, and then performs Metropolis–Hastings correction in sample space. The induced proposal satisfies

$x=g(u)$0

while the acceptance criterion becomes

$x=g(u)$1

The key point is tractability. Direct dependent proposals in sample space would ordinarily require the unknown data density $x=g(u)$2, but the latent-space change of variables cancels those terms, leaving a ratio that depends only on the latent proposal, the latent prior, and the discriminator’s density-ratio estimate. When $x=g(u)$3, REP-GAN reduces to MH-GAN, so the method generalizes independent-proposal sampling to dependent proposal sampling through reparameterization.

The latent proposal can be made gradient-based through a Langevin step,

$x=g(u)$4

which the paper rewrites using the discriminator and the prior. This is not repulsive sampling in the determinantal-process sense. The paper states that the “repulsive” aspect is indirect: dependent local proposals improve acceptance and local exploration, and in multi-modal synthetic experiments REP-GAN preserves more diversity than DDLS. It reports that MH-GAN can have average acceptance below $x=g(u)$5 on CIFAR-10, whereas REP-GAN achieves roughly $x=g(u)$6–$x=g(u)$7, with the MH correction adding only about $x=g(u)$8 extra time compared with REP without MH. On 25-Gaussians and larger mixtures, REP-GAN tends to preserve all modes while avoiding the over-concentration and mode-dropping sometimes seen in purely gradient-based methods (Wang et al., 2021).

4. Shared-randomness contrast and variance reduction for discrete variables

A different use of the term arises in "Reparameterization trick for discrete variables" (Tokui et al., 2016). The paper explicitly does not frame the method as repulsive in the usual sense of pushing samples apart or using negative interactions. Instead, it bypasses discontinuity by marginalizing out the target discrete variable and keeping the remaining randomness reparameterized. For a directed graphical model with objective

$x=g(u)$9

the standard pathwise trick fails because a discrete sample requires a discontinuous threshold or argmax-like map. The paper therefore uses

$\mathbb{R}_+^n$0

and then turns the inner expectation into an exact sum over the configurations of $\mathbb{R}_+^n$1: $\mathbb{R}_+^n$2

For a Bernoulli variable, this yields a difference-of-configurations estimator of the form

$\mathbb{R}_+^n$3

where both $\mathbb{R}_+^n$4 and $\mathbb{R}_+^n$5 are evaluated under the same $\mathbb{R}_+^n$6. The paper interprets this as an application of common random numbers. Its core variance mechanism is

$\mathbb{R}_+^n$7

Shared noise increases the covariance term and reduces the variance of the difference.

The paper proves that the estimator’s variance is no larger than that of the likelihood-ratio method with any baseline, including the optimal input-dependent baseline: $\mathbb{R}_+^n$8 This is the strongest formal guarantee among the cited variance-reduction constructions. On variational learning of sigmoid belief networks, the paper reports that the gradient variance is dramatically smaller, by factors of roughly $\mathbb{R}_+^n$9 to $$g(u)=\exp(u) \quad\text{or}\quad g(u)=\log\cosh(u),$$0, that the proposed estimator learns faster than LR with the adaptive baseline, and that the empirical slowdown on an NVIDIA TITAN X was less than twofold for models with $$g(u)=\exp(u) \quad\text{or}\quad g(u)=\log\cosh(u),$$1. In this literature, any use of “repulsive” is therefore metaphorical: the method contrasts clamped configurations under shared randomness rather than introducing explicit sample repulsion (Tokui et al., 2016).

5. Exact constraints and orthogonal nuisance coordinates in scientific computing

Scientific ML uses reparameterization to remove pathological coupling at the level of boundary conditions or posterior geometry. In "Physics-informed Neural Network: The Effect of Reparameterization in Solving Differential Equations" (Nand et al., 2023), reparameterization means modifying the neural-network output so that the boundary conditions are satisfied exactly by construction: $$g(u)=\exp(u) \quad\text{or}\quad g(u)=\log\cosh(u),$$2 The paper contrasts this with penalty functions, which leave the network output unconstrained and minimize PDE residuals plus weighted boundary-condition residuals. For a one-dimensional bar with

$$g(u)=\exp(u) \quad\text{or}\quad g(u)=\log\cosh(u),$$3

the reparameterized ansatz is

$$g(u)=\exp(u) \quad\text{or}\quad g(u)=\log\cosh(u),$$4

whereas the non-reparameterized case uses $$g(u)=\exp(u) \quad\text{or}\quad g(u)=\log\cosh(u),$$5 and penalty coefficients $$g(u)=\exp(u) \quad\text{or}\quad g(u)=\log\cosh(u),$$6. For a simply supported beam governed by

$$g(u)=\exp(u) \quad\text{or}\quad g(u)=\log\cosh(u),$$7

with

$$g(u)=\exp(u) \quad\text{or}\quad g(u)=\log\cosh(u),$$8

the paper uses the partial reparameterization

$$g(u)=\exp(u) \quad\text{or}\quad g(u)=\log\cosh(u),$$9

It reports approximation errors of $$J_g(u)=\operatorname{diag}(e^{u_i})=\operatorname{diag}(x_i).$$0 versus $$J_g(u)=\operatorname{diag}(e^{u_i})=\operatorname{diag}(x_i).$$1 for the bar problem, and $$J_g(u)=\operatorname{diag}(e^{u_i})=\operatorname{diag}(x_i).$$2 versus $$J_g(u)=\operatorname{diag}(e^{u_i})=\operatorname{diag}(x_i).$$3 for the beam problem. The partially reparameterized beam has zero deviation at the beam ends, whereas the penalty-function case does not satisfy boundary conditions exactly, and the resulting boundary errors propagate into the interior solution, especially in the first and last $$J_g(u)=\operatorname{diag}(e^{u_i})=\operatorname{diag}(x_i).$$4 of the beam.

A related but statistically distinct use appears in "Reducing nuisance prior sensitivity via non-linear reparameterization, with application to EFT analyses of large-scale structure" (Paradiso et al., 2024). There the parameter vector is

$$J_g(u)=\operatorname{diag}(e^{u_i})=\operatorname{diag}(x_i).$$5

with cosmological parameters $$J_g(u)=\operatorname{diag}(e^{u_i})=\operatorname{diag}(x_i).$$6 and nuisance parameters $$J_g(u)=\operatorname{diag}(e^{u_i})=\operatorname{diag}(x_i).$$7. The paper argues that nuisance-prior sensitivity is often a problem of parameterization rather than only prior choice. It first presents the linear residualization

$$J_g(u)=\operatorname{diag}(e^{u_i})=\operatorname{diag}(x_i).$$8

and then the GAM-based nonlinear version

$$J_g(u)=\operatorname{diag}(e^{u_i})=\operatorname{diag}(x_i).$$9

The transformed nuisance variables are approximately orthogonal to the cosmological block in the likelihood geometry. In 100 mock LSS datasets, the paper compares PTB, LM-TB, and GAM-TB, and reports that both transformed bases reduce projection effects, that the transformed cosmological marginals are much more stable, that the MAP distributions across 100 realizations show lower bias, and that best-fit reconstructed monopole and quadrupole power spectra remain consistent with the input spectra with very small $\dot x = -\operatorname{diag}(x)\,\nabla \Phi(x),$0 differences. This suggests a broader notion of repulsive reparameterization: the transformation pushes nuisance directions away from the target-parameter subspace rather than away from a geometric boundary (Paradiso et al., 2024).

6. Stable pathwise optimization and integral reparameterization

Reparameterization also serves as a stabilization layer in differentiable control and rendering, although the repulsive effect is again indirect. "Reparameterization Proximal Policy Optimization" (Zhong et al., 8 Aug 2025) addresses the instability of reparameterization policy gradients in long-horizon or non-smooth environments. For a Gaussian policy,

$\dot x = -\operatorname{diag}(x)\,\nabla \Phi(x),$1

the RPG estimator is

$\dot x = -\operatorname{diag}(x)\,\nabla \Phi(x),$2

RPO imports PPO-style stable sample reuse into this pathwise setting by optimizing a clipped surrogate objective tailored for RPG,

$\dot x = -\operatorname{diag}(x)\,\nabla \Phi(x),$3

together with KL regularization

$\dot x = -\operatorname{diag}(x)\,\nabla \Phi(x),$4

The full policy objective is

$\dot x = -\operatorname{diag}(x)\,\nabla \Phi(x),$5

The algorithm collects short-horizon rollouts, runs BPTT once per trajectory to cache action-gradients $\dot x = -\operatorname{diag}(x)\,\nabla \Phi(x),$6, performs multiple policy epochs with importance-ratio clipping and inverse reparameterization, and updates a double-critic value function using TD-$\dot x = -\operatorname{diag}(x)\,\nabla \Phi(x),$7 regression. The paper states that clipping alone is not enough for stable RPG training, so the KL term is essential. It evaluates RPO on Hopper, Ant, Anymal, Humanoid, and Hand Reorient, reports superior sample efficiency and strong performance, and gives sample-efficiency improvements such as reaching comparable Hand Reorient performance about 3 million steps earlier than SAPO and reaching Hopper score 5000 about 5 million steps sooner than SAPO. Here reparameterization is not boundary repulsion; it is a pathwise mechanism whose updates are constrained by clipping and KL control.

"Neural BRDF Importance Sampling by Reparameterization" (Wu et al., 13 May 2025) pushes the same idea into Monte Carlo rendering. Instead of learning an invertible sampler, it treats importance sampling as learning a substitution rule for the BRDF integral: $\dot x = -\operatorname{diag}(x)\,\nabla \Phi(x),$8 with training objective

$\dot x = -\operatorname{diag}(x)\,\nabla \Phi(x),$9

The rendering estimator becomes

$x_i\to 0$0

To avoid zero BRDF values or zero Jacobians, the paper introduces a defensive relaxed loss with $x_i\to 0$1 and an identity-like map covering the hemisphere. It explicitly states that the method is not framed as explicitly repulsive, but that the support-coverage requirement of the learned map and the defensive mixture help prevent collapse or missed regions. Empirically, it reports the best variance reduction in neural BRDF renderings while maintaining high inference speeds, and states that it is often $x_i\to 0$2–$x_i\to 0$3 faster than analytic mixtures on highly specular materials at equal variance. A plausible implication is that repulsive reparameterization, in this setting, means preserving reachable support under a forward transport rather than penalizing concentration directly (Wu et al., 13 May 2025).