Reinforce Adjoint Matching (RAM)

• RAM is a regression-based RL post-training algorithm for diffusion and flow-matching models that tilts the clean endpoint distribution with a scalar reward while preserving analytic noising.
• It employs a closed-form, reward-corrected target from KL-regularized control, avoiding costly SDE rollouts, backward sweeps, and reward gradients.
• RAM achieves high efficiency by reaching peak rewards in 34–50× fewer training steps, demonstrating scalability on text-to-image alignment tasks.

Searching arXiv for the cited RAM and related adjoint-matching papers. Reinforce Adjoint Matching (RAM) is a regression-based algorithm for RL post-training of diffusion and flow-matching models under KL-regularized reward maximization. In the formulation introduced for text-to-image alignment, the pretrained model’s clean-endpoint distribution is tilted by a scalar reward while the analytic noising law is left unchanged; training therefore consists of drawing a clean endpoint from the current model, evaluating its reward, noising it as in pretraining, and regressing the model against a closed-form reward-corrected target. The method is presented as avoiding SDE rollouts during training, backward adjoint sweeps, and reward gradients, while retaining the supervised-regression structure associated with large-scale pretraining (Bergmeister et al., 11 May 2026).

1. Terminology and research lineage

The expression “adjoint matching” denotes a family of stochastic-control and surrogate-modeling techniques rather than a single algorithmic object. Within that family, the specific term Reinforce Adjoint Matching (RAM) refers to the 2026 RL post-training method for diffusion and flow-matching models (Bergmeister et al., 11 May 2026). Closely related nomenclature appears elsewhere in the literature: the 2024 reward fine-tuning paper describes Reward-Augmented Adjoint Matching (RAM) for continuous-time generative SDEs (Domingo-Enrich et al., 2024); the 2025 sampling paper develops Adjoint Sampling and a replay-buffered RAM loss for learning diffusion samplers from unnormalized energies (Havens et al., 16 Apr 2025); the 2026 maximum-entropy RL paper presents a Reciprocal Adjoint Matching construction, called AMDP in that work, for diffusion policies (Thilges et al., 21 Jun 2026); and the 2021 4D-Var paper uses adjoint matching for neural-network surrogates in data assimilation (Chennault et al., 2021).

Method or usage	Domain	Distinguishing feature
Reinforce Adjoint Matching	RL post-training of diffusion and flow-matching models	Reward-corrected consistency loss with analytic noising
Reward-Augmented Adjoint Matching	Fine-tuning continuous-time generative SDEs	Memoryless stochastic optimal control
Adjoint Sampling / RAM loss	Sampling from unnormalized densities	Replay buffer and closed-form backward kernel
AMDP / Reciprocal Adjoint Matching	Maximum-entropy RL with diffusion policies	Simulation-free actor updates from bridge sampling
Adjoint-matching surrogates	4D-Var data assimilation	Forward and adjoint loss coupling

This terminology overlap matters because the shared phrase “adjoint matching” can obscure substantive differences in objective, state variable, and computational pathway. In the 2026 RL post-training setting, RAM is not a generic policy-gradient estimator; it is a one-step regression objective derived from a KL-regularized control problem (Bergmeister et al., 11 May 2026).

2. KL-regularized control formulation

The 2026 RAM formulation begins from the canonical post-training objective

$$\max_{p}\;\Big\{ \mathbb{E}_{x\sim p}[\,r(x)\,] \;-\;\mathrm{KL}\bigl(p \,\|\,p_0^{\mathrm{ref}}\bigr)\Big\},$$

whose unique solution is the tilted endpoint distribution

$$p_0^*(x)\;\propto\;p_0^{\mathrm{ref}}(x)\,\exp\!\bigl(r(x)\bigr).$$

Here the KL term is described as keeping the new model close to pretraining and preventing out-of-support samples, while the reward tilt shifts mass toward high-reward endpoints (Bergmeister et al., 11 May 2026).

Rather than sampling directly from $p_0^*$, RAM introduces a drift correction $u_t(x)$ to the pretrained model’s backward SDE. The reference backward dynamics are written

$$dX_t \;=\;\bigl(b_t^{\mathrm{ref}}(X_t)\bigr)\,dt\;+\;\sigma_t\,dB_t,\quad X_1\sim\mathcal N(0,I),$$

with

$$b_t^{\mathrm{ref}}(x)=\kappa_t x-\sigma_t^2\nabla_x\log p_t^{\mathrm{ref}}(x),$$

and the controlled process is

$$dX_t^u = \bigl(b_t^{\mathrm{ref}}(X_t^u)+\sigma_t\,u_t(X_t^u)\bigr)\,dt +\sigma_t\,dB_t,\quad X_1^u\sim\mathcal N(0,I).$$

Via Girsanov, maximizing terminal reward minus control energy,

$$\max_{u}\; \mathbb{E}\Bigl[r(X_0^u)\;-\;\tfrac12\!\int_0^1\|u_\tau(X_\tau^u)\|^2\,d\tau\Bigr],$$

is equivalent to the KL-regularized objective in path space (Bergmeister et al., 11 May 2026).

A central structural statement is that the optimal process changes the clean-endpoint law but not the conditional noising law: the endpoint law becomes $p_0^*$, whereas the conditional trajectory law given $X_0$ remains the pretrained Gaussian corruption kernel (Bergmeister et al., 11 May 2026). This claim is the immediate bridge from stochastic optimal control to a pretraining-like regression loss.

A related but distinct 2024 formulation of reward fine-tuning also casts the problem as stochastic optimal control, with optimal control satisfying

$$p_0^*(x)\;\propto\;p_0^{\mathrm{ref}}(x)\,\exp\!\bigl(r(x)\bigr).$$0

but proves that exact target tilting requires a specific memoryless schedule $$p_0^*(x)\;\propto\;p_0^{\mathrm{ref}}(x)\,\exp\!\bigl(r(x)\bigr).$$1 within its SDE family (Domingo-Enrich et al., 2024). The two results concern related SOC views of generative fine-tuning, but not identical setups.

3. Adjoint decomposition and the RAM consistency loss

RAM defines the time-$$p_0^*(x)\;\propto\;p_0^{\mathrm{ref}}(x)\,\exp\!\bigl(r(x)\bigr).$$2 value function and its spatial gradient as

$$p_0^*(x)\;\propto\;p_0^{\mathrm{ref}}(x)\,\exp\!\bigl(r(x)\bigr).$$3

A verification argument yields the optimality condition

$$p_0^*(x)\;\propto\;p_0^{\mathrm{ref}}(x)\,\exp\!\bigl(r(x)\bigr).$$4

The algorithm therefore reduces to obtaining a tractable estimator or proxy for the adjoint $$p_0^*(x)\;\propto\;p_0^{\mathrm{ref}}(x)\,\exp\!\bigl(r(x)\bigr).$$5 (Bergmeister et al., 11 May 2026).

The paper then gives an exact REINFORCE-style decomposition: $$p_0^*(x)\;\propto\;p_0^{\mathrm{ref}}(x)\,\exp\!\bigl(r(x)\bigr).$$6 In practice, the path-cost term is dropped, producing what the paper calls a low-variance reward-only proxy that is exact at initialization and at the optimum (Bergmeister et al., 11 May 2026). This is a methodological simplification rather than a claim that the second term is identically zero.

For linear noising,

$$p_0^*(x)\;\propto\;p_0^{\mathrm{ref}}(x)\,\exp\!\bigl(r(x)\bigr).$$7

the Bayes bridge score is

$$p_0^*(x)\;\propto\;p_0^{\mathrm{ref}}(x)\,\exp\!\bigl(r(x)\bigr).$$8

where $$p_0^*(x)\;\propto\;p_0^{\mathrm{ref}}(x)\,\exp\!\bigl(r(x)\bigr).$$9 is the pretrained velocity model (Bergmeister et al., 11 May 2026). Substituting the reward-only adjoint proxy into the control-matching condition and using the control–velocity relation $p_0^*$0 yields the one-step RAM regression objective

$p_0^*$1

The sampling law inside this loss is explicitly specified: $p_0^*$2, $p_0^*$3, $p_0^*$4, and $p_0^*$5 is drawn from a biasing density such as $p_0^*$6 on $p_0^*$7 (Bergmeister et al., 11 May 2026). Because the noising kernel is analytic, each endpoint can be reused to generate $p_0^*$8 conditionally independent training states.

4. Training procedure and implementation regime

A RAM training iteration is described as follows. First, sample a clean endpoint $p_0^*$9 from the current model $u_t(x)$0 using an off-the-shelf ODE sampler. Second, evaluate the endpoint reward $u_t(x)$1. Third, for $u_t(x)$2 in parallel, sample $u_t(x)$3 and $u_t(x)$4, form

$u_t(x)$5

and compute the closed-form target

$u_t(x)$6

Finally, update $u_t(x)$7 by minimizing

$u_t(x)$8

No SDE rollout, no backward sweep, and no reward gradient appear in this procedure (Bergmeister et al., 11 May 2026).

For the Stable Diffusion 3.5M experiments, the reported settings are highly specific. The adaptation uses LoRA adaptation rank 32, scaling 64. The per-step prompt count is 48; samples per prompt are 24; and targets per image $u_t(x)$9 are 8, giving an effective batch size of $$dX_t \;=\;\bigl(b_t^{\mathrm{ref}}(X_t)\bigr)\,dt\;+\;\sigma_t\,dB_t,\quad X_1\sim\mathcal N(0,I),$$0 training pairs. The sampler during training is 20-step Euler with classifier-free guidance scale 2.0. The sampler at evaluation is 40-step Euler with guidance 4.5 (GenEval & OCR) or 2.0 (PickScore). Optimization uses AdamW, lr=$$dX_t \;=\;\bigl(b_t^{\mathrm{ref}}(X_t)\bigr)\,dt\;+\;\sigma_t\,dB_t,\quad X_1\sim\mathcal N(0,I),$$1, weight decay=$$dX_t \;=\;\bigl(b_t^{\mathrm{ref}}(X_t)\bigr)\,dt\;+\;\sigma_t\,dB_t,\quad X_1\sim\mathcal N(0,I),$$2, and $$dX_t \;=\;\bigl(b_t^{\mathrm{ref}}(X_t)\bigr)\,dt\;+\;\sigma_t\,dB_t,\quad X_1\sim\mathcal N(0,I),$$3. Reward normalization uses group-relative advantage (groups of 24) together with global standard-deviation scaling. The reward coefficients (post-normalization) are 100 for GenEval/OCR and 1000 for PickScore. An exponential moving average of parameters is used with decay=0.9, warmup 0.01 during training, and final EMA for eval (Bergmeister et al., 11 May 2026).

The resulting training pipeline closely mirrors pretraining in one crucial respect: clean endpoints are corrupted analytically and supervision is presented as a closed-form regression target. This suggests that the paper’s scalability claim rests less on a new sampler and more on preservation of the data-generation geometry characteristic of diffusion and flow-matching pretraining.

5. Empirical performance and efficiency

The reported empirical evaluation targets three text-to-image alignment tasks: compositional image generation with GenEval reward, visual text rendering with OCR reward, and human-preference alignment with PickScore reward (Bergmeister et al., 11 May 2026). For each task, the paper reports held-out training rewards and DrawBench image-quality metrics.

On compositional image generation, the pretrained SD3.5M baseline records GenEval 0.64. Flow-GRPO at $$dX_t \;=\;\bigl(b_t^{\mathrm{ref}}(X_t)\bigr)\,dt\;+\;\sigma_t\,dB_t,\quad X_1\sim\mathcal N(0,I),$$4 steps reaches GenEval 0.95. DiffusionNFT at 900 steps also reaches 0.95. RAM at 270 steps reaches GenEval 0.97, with Aesthetic 5.38, DeQA 4.09, ImgRwd 1.19, HPSv2 0.29, and PickScore 22.52; the table marks RAM as best on GenEval, Aesthetic, ImgRwd, HPSv2, and PickScore, and second-best on DeQA (Bergmeister et al., 11 May 2026).

On visual text rendering, Flow-GRPO at 1 200 steps reaches OCR 0.92. AWM at 200 steps reaches OCR 0.97. DiffusionNFT at 100 steps reaches OCR 0.96. RAM at 60 steps reaches OCR 0.97, with Aesthetic 5.23, DeQA 3.90, ImgRwd 0.44, HPSv2 0.26, and PickScore 21.83; the table marks RAM as tied for best on OCR and second-best on the listed quality metrics (Bergmeister et al., 11 May 2026).

On human-preference alignment, Flow-GRPO at $$dX_t \;=\;\bigl(b_t^{\mathrm{ref}}(X_t)\bigr)\,dt\;+\;\sigma_t\,dB_t,\quad X_1\sim\mathcal N(0,I),$$5 steps reaches PickScore 23.31. AWM at 1 000 steps reaches 23.39. DiffusionNFT at 1 400 steps reaches 23.29. RAM at 300 steps reaches PickScore 23.67, with Aesthetic 6.11, DeQA 4.17, ImgRwd 1.36, HPSv2 0.32, and PickScore 23.95 in the quality table; RAM is marked best on PickScore, ImgRwd, HPSv2, and PickScore again in the metric panel, with DeQA second-best (Bergmeister et al., 11 May 2026).

The efficiency claim is explicit. RAM is reported to match Flow-GRPO’s peak GenEval reward (0.95) in $$dX_t \;=\;\bigl(b_t^{\mathrm{ref}}(X_t)\bigr)\,dt\;+\;\sigma_t\,dB_t,\quad X_1\sim\mathcal N(0,I),$$6 fewer training steps; on OCR it reaches Flow-GRPO’s peak (0.92) in $$dX_t \;=\;\bigl(b_t^{\mathrm{ref}}(X_t)\bigr)\,dt\;+\;\sigma_t\,dB_t,\quad X_1\sim\mathcal N(0,I),$$7 fewer steps; and on PickScore it reaches Flow-GRPO’s peak in $$dX_t \;=\;\bigl(b_t^{\mathrm{ref}}(X_t)\bigr)\,dt\;+\;\sigma_t\,dB_t,\quad X_1\sim\mathcal N(0,I),$$8 fewer steps. The paper further states that per-step compute is comparable—for GenEval, 0.66 vs 0.70 GPU-hours—so the wall-clock reduction scales by roughly the same factor (Bergmeister et al., 11 May 2026).

6. Related adjoint-matching variants, misconceptions, and scope boundaries

A recurrent misconception is that RL post-training of diffusion or flow-matching models necessarily requires trajectory-level policy gradients or sample-likelihood surrogates. The 2026 RAM paper directly contrasts itself with three classes of prior methods: policy-gradient methods such as Flow-GRPO, which treat denoising as an MDP and require stochastic SDE rollouts; adjoint-based methods such as Adjoint Matching and ELEGANT, which estimate the adjoint by backpropagating through the entire SDE trajectory; and surrogate-loss approaches such as DiffusionNFT and AWM, which replace the intractable sample likelihood with ELBO surrogates or contrastive positive/negative guidance (Bergmeister et al., 11 May 2026). RAM’s stated distinction is that its targets come from an analytic Bayes bridge score applied to on-policy endpoints.

A second source of confusion is the acronym itself. The 2024 Reward-Augmented Adjoint Matching paper uses adjoint matching for reward fine-tuning of continuous-time generative SDEs and proves a sharp necessity result: to guarantee exact terminal tilting, fine-tuning must use the memoryless schedule $$dX_t \;=\;\bigl(b_t^{\mathrm{ref}}(X_t)\bigr)\,dt\;+\;\sigma_t\,dB_t,\quad X_1\sim\mathcal N(0,I),$$9 (Domingo-Enrich et al., 2024). By contrast, the 2026 Reinforce Adjoint Matching paper emphasizes that, under its KL-regularized optimum, the clean-endpoint distribution is tilted while the pretrained Gaussian noising law is unchanged (Bergmeister et al., 11 May 2026). These are related control-theoretic perspectives, but they are not interchangeable statements.

A third distinction concerns adjacent RAM-like algorithms outside image post-training. Adjoint Sampling alternates an outer endpoint simulation and energy-gradient computation with many inner regression updates from cached $$b_t^{\mathrm{ref}}(x)=\kappa_t x-\sigma_t^2\nabla_x\log p_t^{\mathrm{ref}}(x),$$0 pairs, making $$b_t^{\mathrm{ref}}(x)=\kappa_t x-\sigma_t^2\nabla_x\log p_t^{\mathrm{ref}}(x),$$1–$$b_t^{\mathrm{ref}}(x)=\kappa_t x-\sigma_t^2\nabla_x\log p_t^{\mathrm{ref}}(x),$$2 inner updates per outer simulation/energy evaluation possible; its theoretical summary states that no importance reweighting or off-policy correction is needed, and its experiments report 50–100$$b_t^{\mathrm{ref}}(x)=\kappa_t x-\sigma_t^2\nabla_x\log p_t^{\mathrm{ref}}(x),$$3 more gradient updates per wall-clock in the runtime analysis (Havens et al., 16 Apr 2025). In maximum-entropy RL with diffusion policies, the AMDP method uses reciprocal adjoint matching together with error-function squashing, trust-region policy update, and temperature auto-tuning, and reports update-time behavior such as AMDP (16 diffusion steps) – update: 994 ms versus Reverse-KL chain-backprop (16 steps) – update: 9 944 ms (Thilges et al., 21 Jun 2026). In data assimilation, adjoint matching instead augments surrogate learning with Jacobian or adjoint-vector losses and, on Lorenz-63 4D-Var, the Adj surrogate attains $$b_t^{\mathrm{ref}}(x)=\kappa_t x-\sigma_t^2\nabla_x\log p_t^{\mathrm{ref}}(x),$$4 RMSE, close to Exact 0.83±0.03, at $$b_t^{\mathrm{ref}}(x)=\kappa_t x-\sigma_t^2\nabla_x\log p_t^{\mathrm{ref}}(x),$$5 speedup (Chennault et al., 2021).

Taken together, these works define adjoint matching as a broader design principle: replace end-to-end differentiation through a controlled dynamical process with a regression target derived from first-order optimality and a tractable adjoint proxy. In Reinforce Adjoint Matching specifically, that principle is specialized to KL-regularized RL post-training of diffusion and flow-matching models, where the analytic corruption process inherited from pretraining supplies the bridge between reward evaluation at clean endpoints and scalable gradient updates (Bergmeister et al., 11 May 2026).