Implicit Reparameterization Techniques

• Implicit Reparameterization Techniques are methods that extend the classic reparameterization trick by using implicit differentiation to compute gradients for distributions without closed-form inverses.
• They leverage CDF standardization, acceptance-rejection sampling, and marginalization to overcome high variance in gradient estimation for non-location-scale distributions.
• These techniques enhance applications in variational inference, generative modeling, and reinforcement learning by significantly reducing gradient variance and improving convergence.

Implicit reparameterization techniques constitute a class of pathwise gradient estimators that extend the applicability of the classic reparameterization trick to a wide variety of distributions and sampling procedures, including those for which no analytic inverse transform or straightforward differentiable mapping exists. By leveraging implicit differentiation, marginalization, or auxiliary transformations, these methods enable low-variance, unbiased gradient estimation in settings central to modern latent variable models, Bayesian inference, generative modeling, and reinforcement learning.

1. Foundations and General Principles

Given a family of random variables $x \sim p(x;\theta)$ and differentiable function $f(x)$, the classic reparameterization trick expresses gradients of expectations,

$$\nabla_\theta \mathbb{E}_{x \sim p(x; \theta)}[f(x)],$$

as pathwise derivatives via a transformation $x = g(\varepsilon; \theta)$ where $\varepsilon \sim q(\varepsilon)$ is independent of $\theta$. For many distributions, such as the normal, this "explicit" transformation exists. However, numerous families (e.g., gamma, beta, Dirichlet, von Mises, distributions via accept-reject or MCMC sampling) lack a closed-form inverse, making the classic trick inapplicable.

Implicit reparameterization identifies a standardization function $S(x, \theta) = \varepsilon$ such that sampling $x \sim p(x; \theta)$ is equivalent to drawing $\varepsilon$ from a fixed distribution and mapping to $x$ via $f(x)$0. Implicit differentiation then yields

$f(x)$1

and the pathwise gradient estimator: $f(x)$2 In the univariate case with $f(x)$3 (the CDF), this simplifies to

$f(x)$4

This formulation enables reparameterization-based gradient estimation for all continuous distributions with tractable and differentiable CDFs, regardless of invertibility in closed form (Figurnov et al., 2018).

2. Applications to Non-Location-Scale Distributions

Gamma, Beta, and Dirichlet Distributions

For the gamma distribution $f(x)$5, sampling is not amenable to the explicit trick. By considering the standardization $f(x)$6 (unit-rate gamma CDF), one computes gradients via implicit differentiation: $f(x)$7 where $f(x)$8 is the PDF. Beta gradients exploit the representation $f(x)$9 with $$\nabla_\theta \mathbb{E}_{x \sim p(x; \theta)}[f(x)],$$0, $$\nabla_\theta \mathbb{E}_{x \sim p(x; \theta)}[f(x)],$$1, and chain rule. Dirichlet gradients generalize this approach for simplex-valued variables (Figurnov et al., 2018).

These techniques bypass the high variance and bias associated with score-function (likelihood-ratio) estimators and enable efficient variational inference and latent variable modeling with gamma and Dirichlet posteriors.

von Mises and Other Complex Families

For distributions such as von Mises, the absence of a tractable inverse CDF is overcome by choosing $$\nabla_\theta \mathbb{E}_{x \sim p(x; \theta)}[f(x)],$$2 as the CDF and computing implicit derivatives accordingly (Figurnov et al., 2018).

3. Implicit Reparameterization through Acceptance-Rejection Sampling

Many random variable simulators (e.g., for gamma, truncated, or compound distributions) use accept-reject algorithms, introducing discontinuities that preclude the explicit trick. The "reparameterization through acceptance-rejection sampling" method constructs the joint density over proposal variables and accept/reject flags, marginalizes the indicator, and derives a pathwise gradient: $$\nabla_\theta \mathbb{E}_{x \sim p(x; \theta)}[f(x)],$$3 where $$\nabla_\theta \mathbb{E}_{x \sim p(x; \theta)}[f(x)],$$4 is the transformation mapping proposal $$\nabla_\theta \mathbb{E}_{x \sim p(x; \theta)}[f(x)],$$5 to $$\nabla_\theta \mathbb{E}_{x \sim p(x; \theta)}[f(x)],$$6, and $$\nabla_\theta \mathbb{E}_{x \sim p(x; \theta)}[f(x)],$$7 the marginal density of accepted $$\nabla_\theta \mathbb{E}_{x \sim p(x; \theta)}[f(x)],$$8 (Naesseth et al., 2016).

Empirically, this estimator achieves orders-of-magnitude lower gradient variance than score-function or generalized reparameterization methods, enabling stochastic variational inference for gamma, Dirichlet, and related families.

4. Marginalization-Based Techniques for Discrete Variables

The classic reparameterization trick fails for discrete latents due to the non-differentiable nature of mappings from continuous noise to discrete outcomes. Marginalization-based techniques circumvent this by analytically summing (or integrating) over discrete configurations conditional on shared random noise for remaining variables: $$\nabla_\theta \mathbb{E}_{x \sim p(x; \theta)}[f(x)],$$9 Common random numbers (CRN) introduce strong covariance across $x = g(\varepsilon; \theta)$0 values, reducing gradient variance. The resulting estimator is unbiased with variance provably no greater than any likelihood-ratio estimator, even with the optimal baseline (Tokui et al., 2016).

Applications to deep sigmoid belief networks substantially decrease gradient variance per layer (by $x = g(\varepsilon; \theta)$1–$x = g(\varepsilon; \theta)$2 versus LR) and improve ELBO convergence in variational learning.

5. Implicit Variational Inference with Markov Chain–Induced Posteriors

Implicit reparameterization can induce variational families by transforming simple base distributions via (possibly learned) parametric maps and then applying MCMC kernels. If $x = g(\varepsilon; \theta)$3, and MCMC (e.g., HMC, Metropolis) is run in $x = g(\varepsilon; \theta)$4-space, this yields an implicit variational family $x = g(\varepsilon; \theta)$5 that can be sampled even if its density cannot be evaluated.

The reparameterized evidence lower bound for $x = g(\varepsilon; \theta)$6 avoids the need for density-ratio estimation, relying instead exclusively on samples and evaluating

$x = g(\varepsilon; \theta)$7

in the expectation (Titsias, 2017). This approach flexibly matches complex, non-Gaussian posteriors, demonstrated on nonlinear latent-variable models and variational autoencoders. As $x = g(\varepsilon; \theta)$8, MCMC yields $x = g(\varepsilon; \theta)$9 and recovers the true posterior.

6. Invariant Statistical Losses and Implicit Generative Modeling

For implicit generative models, a CDF-based statistical loss computes how well the generator $\varepsilon \sim q(\varepsilon)$0, $\varepsilon \sim q(\varepsilon)$1, matches the true data distribution. The probability-integral transform $\varepsilon \sim q(\varepsilon)$2, with

$\varepsilon \sim q(\varepsilon)$3

enables constructing a rank-based discrepancy (e.g., between the empirical ranks of generated vs. observed samples) as a uniformly-invariant loss—termed Invariant Statistical Loss (ISL) (Frutos et al., 2024). The ISL is differentiable, computed without adversarial training, and its gradient flows naturally through the generator.

Empirical results show ISL-based reparameterization achieves or surpasses state-of-the-art generative adversarial nets (GANs) and diffusion models on a range of 1D and temporal density estimation tasks, with stable training and no mode collapse.

7. Reinforcement Learning and Bounded Action Spaces

In policy optimization for reinforcement learning, it is often desirable to use bounded distributions (e.g., Beta, Dirichlet) for actions. The implicit reparameterization gradient computes, for a Beta-distributed action $\varepsilon \sim q(\varepsilon)$4,

$\varepsilon \sim q(\varepsilon)$5

Implementation requires numerically stable differentiation of incomplete beta functions and special functions (e.g., digamma). Empirical studies with Soft Actor-Critic (SAC) using implicit gradients for Beta policies show no loss in sample efficiency or stability compared to squashed Gaussian baselines, and in some environments, improved performance (Libera, 2024).

Table: Principal Implicit Reparameterization Techniques and Contexts

Technique	Targeted Distributions	Primary Reference
Implicit differentiation of CDF/standardization	Gamma, Beta, Dirichlet, vM	(Figurnov et al., 2018)
Acceptance-rejection pathwise gradients	Gamma, Dirichlet, trunc.	(Naesseth et al., 2016)
Marginalization + CRN for discrete latents	Bernoulli, categorical	(Tokui et al., 2016)
Markov chain implicit variational families	Complex posteriors, VAEs	(Titsias, 2017)
Invariant Statistical Loss (rank/CDF)	Implicit generators	(Frutos et al., 2024)
CDF implicit gradient in policy optimization	Beta, Dirichlet (RL)	(Libera, 2024)

Limitations and Implementation Considerations

Implicit reparameterization requires numerically differentiable standardization maps such as CDFs. Distributions lacking tractable CDFs remain challenging unless surrogates or hybrid estimators are used. Numerical stability may require special-function libraries and carefully constrained parameterizations (e.g., for Beta/Dirichlet shape parameters).

In higher dimensions, the multivariate distributional transform may be costly ($\varepsilon \sim q(\varepsilon)$6), though structure or coupling (e.g., copulas) can ameliorate this. For acceptance-rejection approaches, low acceptance rates can incur noisy corrections, mitigated by augmentation or improved proposals (Naesseth et al., 2016).

Empirical Impact and Summary

Across variational inference, generative modeling, and reinforcement learning, implicit reparameterization techniques achieve variance and convergence profiles competitive with, and often superior to, competing estimators. They enable practically unbiased, low-variance gradients for a wide range of distributions previously restricted to score-function or surrogate approximations, significantly broadening the array of tractable latent variable models, flexible posteriors, and action policies (Figurnov et al., 2018, Tokui et al., 2016, Titsias, 2017, Frutos et al., 2024, Libera, 2024, Naesseth et al., 2016).