---
title: 'DEFT: Doob’s h-transform Fine-Tuning'
url: https://www.emergentmind.com/topics/doob-s-h-transform-efficient-fine-tuning-deft
type: topic
---

# DEFT: Doob’s h-transform Fine-Tuning

Searching arXiv for DEFT and closely related Doob h-transform diffusion adaptation papers.
Searching arXiv for "DEFT diffusion Doob h-transform" and related follow-up work.
Doob’s \(h\)-transform efficient fine-tuning (DEFT) is a diffusion-model adaptation framework that reinterprets conditional generation as conditioning an unconditional reverse diffusion process and then learns only the resulting generalized \(h\)-transform correction, rather than retraining the full backbone. In its original formulation, DEFT keeps a pretrained unconditional diffusion or score model fixed and trains a small auxiliary network \(h_t^\phi\) so that the conditional reverse dynamics are realized as the unconditional dynamics plus a learned additive correction. The method was introduced to exploit large, expensively trained unconditional diffusion models for inverse problems and other conditional generation tasks, while unifying conditional training and conditional sampling under Doob’s \(h\)-transform [2406.01781].

## 1. Problem setting and conceptual definition

DEFT is motivated by a standard posterior-sampling setup in which a pretrained unconditional model approximates the prior score
\[
s_t^{\theta^*}(x_t) \approx \nabla_{x_t} \log p_t(x_t),
\]
while the downstream task requires sampling from \(p(x_0\mid y)\), or equivalently from the posterior score \(\nabla_{x_t}\log p_t(x_t\mid y)\) [2406.01781]. Inverse problems supply the canonical examples: inpainting, super-resolution, CT reconstruction, and motif-constrained protein generation all require conditional sampling, but end-to-end conditional retraining is expensive, and inference-time guidance methods are often hyperparameter-sensitive or computationally heavy.

The central DEFT claim is that the missing conditional information is not an arbitrary task-specific residual. It is the generalized \(h\)-transform term induced by conditioning the reverse diffusion process. This makes DEFT conceptually different from ordinary parameter-efficient fine-tuning: instead of partially modifying the unconditional backbone, it keeps that backbone unchanged and learns the time-dependent correction needed to convert the unconditional reverse process into the correct conditional one [2406.01781].

This framing also gives DEFT a precise efficiency target. The expensive pretrained unconditional generator remains intact, and only a small conditional module is optimized. In the original imaging and protein experiments, the trainable network is repeatedly described as much smaller than the base model, often about \(4\%\)–\(9\%\) of the backbone in the major experiments, with about \(6\%\)–\(17\%\) in the CT setting [2406.01781]. A practical implication is that DEFT can be used even when only forward-pass access to the unconditional model is available.

## 2. Doob’s \(h\)-transform and the generalized conditional reverse process

The mathematical basis of DEFT is the reverse-time SDE formulation of diffusion models. The unconditional forward process is written as
\[
dx_t = f_t(x_t)\,dt + \sigma_t\, dW_t, \qquad x_0 \sim p_0,
\]
and the unconditional reverse process as
\[
dx_t = \left(f_t(x_t) - \sigma_t^2 \nabla_{x_t}\ln p_t(x_t)\right)\,dt + \sigma_t\, d\bar W_t, \qquad x_T \sim p_T.
\]
If one conditions on a terminal event \(x_0\in B\), the reverse process remains an SDE but acquires an extra drift term,
\[
dx_t = \left( \bar b_t(x_t) - \sigma_t^2 \nabla_{x_t}\ln \bar p_{0|t}(x_0\in B \mid x_t) \right)\,dt + \sigma_t\, d\bar W_t,
\]
where the added gradient is the classical Doob \(h\)-transform correction [2406.01781].

DEFT uses the generalized version appropriate for noisy observations and posterior sampling. Given an observation model \(p(y\mid x_0)\), the conditional reverse process becomes
\[
dx_t = \left( \bar b_t(x_t) - \sigma_t^2 \nabla_{x_t}\ln p_{y|t}(y\mid x_t) \right)\,dt + \sigma_t\, d\bar W_t,
\]
with
\[
p_{y|t}(y\mid x_t) = \int p(y\mid x_0)\,\bar p_{0|t}(x_0\mid x_t)\,dx_0.
\]
This yields the exact posterior-score decomposition
\[
\nabla_{x_t}\ln p_t(x_t\mid y) = \nabla_{x_t}\ln p_t(x_t) + \nabla_{x_t}\ln p_{y|t}(y\mid x_t),
\]
which is the formal version of “prior score plus guidance” [2406.01781].

The original DEFT paper also gives a path-space representation. If \(\mathbb P\) is the path measure of the unconditional SDE, then the conditional path measure satisfies
\[
\mathbb Q^* = \frac{p(x_0\mid y)}{p_0(x_0)}\,\mathbb P.
\]
From this viewpoint, denoising score matching, stochastic control, and Schrödinger-bridge-style formulations are equivalent representations of the same \(h\)-transformed conditional process. This equivalence is one of DEFT’s main theoretical unifications: conditional sampling and conditional training target the same object, but parameterize it differently [2406.01781].

## 3. Training objective, learned correction, and inference mechanism

DEFT learns a network \(h_t^\phi(x_t,y)\) whose optimal value is the generalized \(h\)-transform score correction,
\[
h_t^{\phi^*}(x_t,y)=\nabla_{x_t}\ln p_t(y\mid x_t).
\]
The core score-matching objective is
\[
\min_{h^\phi} \mathbb{E}_{\substack{(x_0,y)\sim p(x_0,y)\\ t\sim U(0,T),\, x_t\sim p_{t|0}(x_t|x_0)}} \left[ \left\| \left(h_t^\phi(x_t,y)+\nabla_{x_t}\ln p_t(x_t)\right) - \nabla_{x_t}\ln p_{t|0}(x_t|x_0) \right\|^2 \right].
\]
Its minimizer is exactly the desired conditional correction [2406.01781].

For DDPM-style \(\epsilon\)-parameterization, this becomes
\[
\min_\phi \mathbb{E}_{x_0,y,\epsilon,t} \left[ \left\| \left(h_t^\phi(x_t,y)+\epsilon_t^{\theta^*}(x_t)\right)-\epsilon \right\|^2 \right],
\]
with
\[
x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon, \qquad \epsilon\sim \mathcal N(0,I).
\]
The pretrained unconditional model \(\epsilon_t^{\theta^*}\) is frozen. Only \(\phi\) is updated. A consequential implementation detail is that the objective needs evaluation of the pretrained model but not differentiation through it [2406.01781].

The learned correction also admits a Tweedie-estimate interpretation. If \(\hat x_0(x_t)\) is the unconditional denoised estimate, then the conditional estimate is
\[
\hat x_0(x_t,y) = \hat x_0(x_t) - \frac{\sqrt{1-\bar\alpha_t}}{\sqrt{\bar\alpha_t}}\,h_t^{\phi^*}(x_t,y).
\]
Thus DEFT may be viewed as learning how to correct the unconditional denoiser into a conditional denoiser [2406.01781].

The practical parameterization used in the original paper is
\[
h_t^\phi(x_t,y) = \mathrm{NN}_1^\phi\!\left( x_t,\hat x_0(x_t),\nabla_{\hat x_0}\log p(y\mid \hat x_0(x_t)), t \right) + \mathrm{NN}_2^\phi(t)\,\nabla_{\hat x_0}\log p(y\mid \hat x_0(x_t)).
\]
For imaging, \(\mathrm{NN}_1^\phi\) is an attention U-Net and \(\mathrm{NN}_2^\phi\) is a small 3-layer MLP with SiLU activations; the final layer of \(\mathrm{NN}_1^\phi\) and most of \(\mathrm{NN}_2^\phi\) are zero-initialized so that training starts close to the unconditional sampler [2406.01781].

At inference, DEFT simply replaces the unconditional \(\epsilon\)-prediction by the corrected one,
\[
\hat\epsilon = \hat\epsilon_\theta + \hat\epsilon_\phi,
\]
where \(\hat\epsilon_\theta=\epsilon_t^{\theta^*}(x_t)\) and \(\hat\epsilon_\phi=h_t^\phi(x_t,\hat x_0,y)\). The resulting \(\hat\epsilon\) is then used in the standard DDIM or DDPM reverse update. In operational terms, the backbone remains an unconditional prior model and the \(h\)-network supplies the conditional score correction [2406.01781].

## 4. Position relative to guidance, conditional training, and adjacent \(h\)-transform methods

A major contribution of DEFT is interpretive rather than purely algorithmic. The framework recasts several existing diffusion-conditioning strategies as approximations, special cases, or alternative parameterizations of the same generalized \(h\)-transform [2406.01781].

Reconstruction-guidance methods such as DPS are interpreted as online approximations to \(\nabla_{x_t}\ln p_{y|t}(y\mid x_t)\). In that view, guidance-scale tuning acts as an approximation parameter, and the computational burden comes from repeatedly differentiating through the large unconditional model during sampling. DEFT moves that computation offline into a learned correction network, eliminating test-time backpropagation through the base model and avoiding a guidance-scale schedule [2406.01781].

Methods that directly train the full conditional score, such as conditional denoising estimators, target \(\nabla_{x_t}\log p_t(x_t\mid y)\) itself. DEFT instead learns only the additive residual
\[
\nabla_{x_t}\log p_t(y\mid x_t),
\]
while reusing the unconditional score already stored in the base network. This is why DEFT is parameter-efficient rather than full conditional retraining [2406.01781].

Subsequent literature has developed several closely related \(h\)-transform variants. “Iterative Importance Fine-tuning of Diffusion Models” treats DEFT as prior work and extends it to a self-supervised regime in which target samples are unavailable. It iteratively generates trajectories, reweights them with path-space importance weights, resamples, and refits the \(h\)-transform via a supervised score-matching loss, thereby yielding an iterative, importance-weighted DEFT-like procedure [2502.04468]. “Training-Free Adaptation of Diffusion Models via Doob’s \(h\)-Transform” instead computes the Doob correction online by Monte Carlo during sampling and performs no parameter updates at all, making it an inference-time rather than fine-tuning variant [2602.16198]. “Inference-Time Alignment for Diffusion Models via Doob’s Matching” likewise keeps the base score frozen but learns an auxiliary \(h\)-head by gradient-penalized regression, with non-asymptotic guarantees for both guidance estimation and terminal-sample convergence [2601.06514].

These developments suggest a broader research program around the same decomposition:
\[
\text{target score} = \text{base score} + \text{\(h\)-transform correction}.
\]
A plausible implication is that DEFT functions as the parameter-efficient branch of this family, whereas DOIT and \(h\)-Edit occupy the training-free inference-time branch [2602.16198].

## 5. Empirical results in imaging, medical reconstruction, and protein design

In the original DEFT experiments, the method is evaluated on natural-image inverse problems, CT reconstruction, and protein motif scaffolding, with the explicit claim that it is much faster than existing baselines while achieving state-of-the-art performance across a variety of linear and non-linear benchmarks [2406.01781].

On ImageNet \(256\times256\) inverse problems, DEFT uses a pretrained unconditional model of about \(500\)–\(550\)M parameters and a trainable correction network of about \(23\)M parameters. In inpainting, DEFT reports PSNR \(22.18\), SSIM \(0.85\), KID \(0.29\), LPIPS \(0.09\), and top-1 \(71.7\), while the total timing is \(5.2\) hours including training and \(4.36\) seconds per sample. The paper notes that RED-diff is stronger on distortion metrics in this task, whereas DEFT is best on perceptual metrics [2406.01781].

On super-resolution, DEFT again sacrifices some PSNR but attains the best perceptual metrics, with KID \(1.78\), LPIPS \(0.12\), and top-1 \(71.9\). On HDR reconstruction, it reports PSNR \(28.51\), SSIM \(0.89\), and KID \(0.10\), outperforming the cited baselines. On phase retrieval, the reported values are PSNR \(13.03\), SSIM \(0.32\), and KID \(80.89\). On nonlinear deblurring, DEFT reports PSNR \(25.16\), SSIM \(0.79\), KID \(0.34\), LPIPS \(0.09\), and top-1 \(69.9\), with timing around \(5.2\) hours total and \(4.4\)–\(4.6\) seconds per sample [2406.01781].

In CT, DEFT is compared against DPS and RED-diff on AAPM 2016 and LoDoPab-CT. On AAPM, DEFT reports PSNR \(34.73\), SSIM \(0.887\), and time \(16.3\)s; on LoDoPab-CT, it reports PSNR \(35.81\), SSIM \(0.876\), and time \(13.8\)s. The paper emphasizes that DEFT uses 100 DDIM steps while the baselines use 1000, and describes the medical-imaging results as strong reconstruction performance with substantially lower sampling cost [2406.01781].

In protein motif scaffolding, DEFT fine-tunes a \(200\)k-parameter module on top of a pretrained unconditional Genie model with 4.1M parameters. Reported success rates are \(9.2\%\) for DEFT and \(1.3\%\) for DPS, with DEFT solving \(10/12\) tasks and DPS solving \(5/12\). An amortized conditional model still performs better, at \(24.5\%\), but it uses a larger model and longer training. The paper therefore positions DEFT as the more efficient conditional adaptation scheme rather than the highest-capacity fully amortized one [2406.01781].

## 6. Later instantiations, limitations, and nomenclature

A direct application-specific instantiation is DEFT-VTON, which adapts the DEFT principle to virtual try-on. There, the pretrained autoencoder and score diffusion U-Net are frozen, and only the h-transform network is trained. The paper reports that baseline PEFT trains about \(5.51\)–\(5.52\%\) of the backbone, whereas DEFT trains only about \(1.42\%\). It further adds an adaptive consistency loss to reduce denoising steps, reporting that the consistency-enhanced system can work with as few as 15 denoising steps, that quality metrics stabilize after about 12 steps in an ablation, and that adding the consistency loss speeds up VTO inference by up to \(40\%\). On VITON-HD, the 25-step DSM-only DEFT variant reports \(0.9118\) SSIM, \(0.0533\) LPIPS, \(8.3351\) FID, and \(0.5212\) KID, while the 15-step consistency-enhanced model reports \(0.9130\) SSIM, \(0.0542\) LPIPS, \(8.8567\) FID, and \(1.016\) KID [2509.13506].

DEFT’s main practical limitation in the original formulation is that it requires a small paired fine-tuning dataset for the target task. This is the central tradeoff relative to zero-shot guidance methods such as DPS or \(\Pi\)GDM. Another limitation, made explicit in DEFT-VTON, is dependence on the pretrained unconditional model: because the trainable \(h\)-transform network is only a small fraction of the backbone, artifacts, biases, and failure modes of the base prior can propagate into the adapted model [2406.01781].

The acronym also requires disambiguation. In later literature, “DEFT” may refer to unrelated methods. “Gradients Must Earn Their Influence: Unifying SFT with Generalized Entropic Objectives” defines DEFT as Dynamic Entropy Fine-Tuning and explicitly does not discuss Doob’s \(h\)-transform [2602.11424]. “Device-Edge Cooperative Fine-Tuning of Foundation Models as a 6G Service” uses DEFT to mean device-edge fine-tuning, again without any Doob-transform content [2310.18602]. Within diffusion-model adaptation, however, DEFT specifically denotes the frozen-backbone, learned-\(h\)-correction framework introduced for conditional generation in inverse problems [2406.01781].

Taken together, the DEFT line establishes a particular design pattern for diffusion adaptation: preserve the pretrained unconditional prior, learn only the generalized \(h\)-transform correction, and realize conditional generation by adding that correction to the reverse denoising dynamics. Subsequent work suggests that this pattern extends both toward self-supervised fine-tuning and toward training-free inference-time control, but the defining DEFT identity remains the same:
\[
\text{conditional score} = \text{unconditional score} + \text{learned generalized \(h\)-transform}.
\]

Source: https://www.emergentmind.com/topics/doob-s-h-transform-efficient-fine-tuning-deft