Tied Diffusion Guidance (TDG)

• TDG is a guided-diffusion paradigm where a learned guidance module steers the denoising process toward higher-value actions.
• In offline RL, the guidance network biases the diffusion model with classifier-free reward guidance, highlighting both joint-training challenges and benefits.
• In text-to-image generation, TDG incorporates a weak conditional branch to expose evaluation pitfalls while maintaining semantic alignment.

Searching arXiv for the cited papers and closely related guidance literature. [arXiv search] Query: (Xie et al., 26 Feb 2026) Tied Diffusion Guidance (TDG) denotes a guided-diffusion configuration in which a learned guidance mechanism steers the diffusion sampling trajectory toward higher-value actions, while the guidance and diffusion components are trained together or otherwise remain tightly coupled. In the offline-RL literature summarized by "Modular Diffusion Policy Training: Decoupling and Recombining Guidance and Diffusion for Offline RL," TDG-style methods are the immediate point of comparison for a later modular alternative that argues such coupling is unnecessary when guidance depends only on fixed offline tuples $\mathcal{D}=\{s,a,r,s'\}$ (Chen et al., 19 May 2025). The acronym TDG is also used, in a different subfield, for "Transcendent Diffusion Guidance" in text-to-image generation, where a weak conditional branch derived from a corrupted prompt is introduced to expand the guidance geometry and, more importantly, to demonstrate an evaluation pitfall in conventional preference-based benchmarking (Xie et al., 26 Feb 2026). The shared principle is guided perturbation of denoising dynamics; the substantive differences lie in domain, objective, and the role of coupling.

1. Terminology and scope

In the offline-RL setting, TDG-style methods are described as using a guidance model to steer diffusion sampling toward higher-value actions, with guidance and diffusion components trained together or at least in a tightly coupled way. The key contrast drawn in the cited work is not between diffusion and non-diffusion methods, but between coupled guidance-policy optimization and a later decoupled regime in which the guidance module is learned first, frozen, and then reused during diffusion training and inference (Chen et al., 19 May 2025).

The acronym TDG is overloaded in the cited material. In offline RL, it refers to a tied or tightly coupled guidance-diffusion paradigm. In text-to-image generation, TDG refers specifically to Transcendent Diffusion Guidance, a method that adds a third prediction branch based on a weakened prompt. This distinction is substantive rather than merely terminological: the offline-RL usage concerns value-guided action generation under fixed datasets, whereas the text-to-image usage concerns prompt-conditioned denoising and the evaluation of guidance methods under human-preference metrics (Xie et al., 26 Feb 2026).

Usage of TDG	Domain	Core characterization
Tied Diffusion Guidance	Offline RL	Guidance steers diffusion toward higher-value actions; guidance and diffusion are tightly coupled
Transcendent Diffusion Guidance	Text-to-image generation	Adds a weak conditional branch from a corrupted prompt to modify guidance geometry

A common source of confusion is to treat these as the same method family. The cited material does not support that reading. What it does support is a broader conceptual linkage: both usages rely on modifying the reverse diffusion trajectory with an auxiliary signal, and both become case studies in how guidance should be trained or evaluated.

2. Core mechanism in offline RL

In the offline-RL formulation related to TDG, the guidance module is a standalone value estimator $Q_\phi(s,a)$, while the diffusion model learns the policy $\pi_\theta(a\mid s)$. Guidance is injected during diffusion training and inference using classifier-free style reward guidance (Chen et al., 19 May 2025). This preserves the standard diffusion role of modeling the action manifold while allowing a reward-aware module to bias denoising toward higher-return regions.

The forward noising process is written as

$$a^k = \sqrt{\bar{\alpha}_k} a^0 + \sqrt{1 - \bar{\alpha}_k}\,\epsilon,\quad \epsilon\sim\mathcal N(0,I).$$

The reverse process is then perturbed by gradients from the value estimator. The cited material gives several equivalent guidance expressions, including

$\tilde a^0 = a^0 + \lambda \nabla_a Q_\phi(s,a^0),$

$$\epsilon_\theta(a_t,s,t)\leftarrow \epsilon_\theta(a_t,s,t)+\alpha \nabla_{a_t}Q(s,a_t),$$

and the explicit sampling update

$$a_{t+1}=a_t+\eta_t\cdot \frac{\nabla_a Q_\phi(a_t)}{\|\nabla_a Q_\phi(a_t)\|}+\sqrt{2\tau_t}\xi_t.$$

The intended decomposition of roles is explicit. The diffusion model still learns the data manifold, but the frozen Q-network biases sampling toward high-return regions. In this sense, TDG-style guidance is not itself the policy; it is an external steering signal. This suggests a view of guided diffusion policies as composite systems with separable generative and evaluative components, even if earlier implementations keep them tied during training.

3. Coupled training and its limitations

The main critique of tied guidance-diffusion training in offline RL is that guidance is not uniformly required at all stages. The cited work identifies three observations: no-guidance diffusion can sometimes improve faster initially than guided diffusion, noisy guidance can disrupt learning entirely, and accurate guidance substantially reduces reward variance and improves overall performance (Chen et al., 19 May 2025). The central warning is therefore stage-dependent: guidance is useful only when it is sufficiently accurate.

This issue is especially acute in offline RL because there is no online exploration to recover from early errors in the guidance signal. The paper’s argument is that if the guidance network is inaccurate early in joint training, it can actively mislead the diffusion model. The resulting failure mode is a harmful feedback loop in which an unconverged critic injects unstable directional information into denoising. The cited discussion highlights this sensitivity in TD-based methods such as DQL and EDP, and notes that EDP, because it relies more heavily on one-step denoising, is even more sensitive to the quality of guidance (Chen et al., 19 May 2025).

A common misconception is that stronger guidance should monotonically improve learning. The cited evidence does not support that conclusion. Instead, it supports a conditional claim: guidance quality, training stage, and algorithm choice jointly determine whether guidance helps, is unnecessary, or becomes actively harmful. This is the principal methodological limitation of tied training regimes.

4. Decoupling TDG: guidance-first diffusion training

The modular alternative presented as "Modular Diffusion Policy Training" or "Guidance-First Diffusion Training (GFDT)" is explicitly framed as closely related to TDG in spirit and mechanism, but as pushing the idea further into a fully modular offline-RL training pipeline (Chen et al., 19 May 2025). Its central claim is that, in offline RL, the coupling of guidance and diffusion is unnecessary because the guidance signal depends only on the fixed dataset $(s,a,r,s')$, not on the evolving diffusion policy.

The pipeline has three steps. First, the guidance module is trained independently as a value estimator, typically a Q-function $Q_\phi(s,a)$ learned by temporal-difference learning. Second, the guidance module is frozen. The cited work emphasizes that freezing is crucial because it prevents the guidance from drifting while the diffusion policy is learning and reduces memory overhead, since the guidance module no longer needs to remain in the active training graph. Third, the frozen guidance module is used to bias the diffusion model through classifier-free reward guidance during diffusion training and inference (Chen et al., 19 May 2025).

The paper states that if the guidance network is kept trainable during diffusion training, the gains diminish due to overfitting and self-reinforcing bias. It also gives an informal theoretical sketch based on batch-constrained RL and a balance between reward optimization and behavior prior: $\nabla_\theta \mathcal{L}_{\text{total} = \nabla_\theta \mathbb{E}[Q(s,a)] - \lambda \nabla_\theta \mathcal{L}_{\text{BC}.$ This is not presented as a full formal proof, but as justification for why a pretrained guidance model can accelerate diffusion training while remaining stable, provided that the learned guidance stays within the offline data support.

Relative to tied guidance, the significance of GFDT is conceptual as much as algorithmic. It reframes guidance as a reusable value-estimation module rather than an inseparable part of the policy. This suggests that what had been treated as a joint-training necessity may in fact be an implementation convention.

5. Transferability, variance reduction, and practical consequences

A major empirical finding associated with the modular reinterpretation of TDG is cross-module transferability. The cited work reports two forms of transfer: using one guidance model during training and a different independently trained guidance model during inference, and cross-algorithm transfer in which a guidance module trained with IDQL can be paired with a DQL diffusion model, and vice versa, without additional training (Chen et al., 19 May 2025). The reported outcome is baseline-level performance together with strong modularity and transferability.

The same study reports that applying two independently trained guidance models, one during training and the other during inference, can significantly reduce normalized score variance, including reducing IQR by $Q_\phi(s,a)$0. It also reports around $Q_\phi(s,a)$1 reduction in max variance and a meaningful reduction in median variance. The cited interpretation is that using the same model to guide itself can amplify its own estimation biases, whereas switching to an independently trained guidance network breaks that loop and acts like a target-network-style regularizer, with an explicit stabilizing analogy to Double Q-learning: $Q_\phi(s,a)$2

These results were obtained on 8 PyBullet D4RL tasks with diffusion RL algorithms including DQL, IDQL, and EDP. The empirical patterns reported are that GFDT consistently rises faster in early training, few-shot settings benefit the most, final performance improves or matches baselines, and variance is lower when guidance is decoupled, especially when using a different independently trained guidance module (Chen et al., 19 May 2025).

The cited work also emphasizes practical resource implications. The overall computation may remain similar, but the peak memory footprint is lower because the diffusion training stage no longer needs to backpropagate through the guidance model. This makes the procedure more plug-and-play in offline RL pipelines where multiple seeds, checkpoints, and model variants are commonly evaluated.

6. Transcendent Diffusion Guidance in text-to-image generation

In the text-to-image literature, TDG denotes Transcendent Diffusion Guidance, introduced in "Guidance Matters: Rethinking the Evaluation Pitfall for Text-to-Image Generation" (Xie et al., 26 Feb 2026). Here the motivation is not offline policy optimization but the diagnosis of a benchmarking problem: common human preference models exhibit a strong bias toward large guidance scales, so increasing classifier-free guidance can improve quantitative evaluation scores even when image quality is severely damaged by oversaturation and artifacts.

Transcendent Diffusion Guidance constructs a weak conditional branch by randomly replacing prompt tokens with the empty token $Q_\phi(s,a)$3. If the prompt is

$Q_\phi(s,a)$4

then the weakened prompt $Q_\phi(s,a)$5 is formed by replacing indices $Q_\phi(s,a)$6 as

$Q_\phi(s,a)$7

In practice, the paper states that

$Q_\phi(s,a)$8

so half of the tokens are randomly replaced. The weak conditional score is then

$Q_\phi(s,a)$9

The full TDG update combines unconditional, conditional, and weak conditional predictions: $\pi_\theta(a\mid s)$0 The geometric interpretation stated in the paper is that standard CFG explores the line segment between unconditional and conditional predictions, whereas TDG adds a third, weakly conditioned point so that the sampling directions span a hyperplane rather than a line (Xie et al., 26 Feb 2026).

The paper groups TDG together with SAG, PAG, SEG, CFG++, Z-Sampling, FreeU, and APG as methods that modify guidance trajectories, often by introducing a weak condition term or otherwise increasing the effective guidance scale. The stated significance of TDG in that paper is not that it reliably improves practical image generation, but that it can significantly improve human-preference scores in the conventional evaluation framework while not working in practice.

7. Evaluation pitfalls, misconceptions, and broader significance

The text-to-image TDG paper argues that recent guidance methods may overfit an overlooked large-guidance bias in human preference models. The preference models specifically named are HPS v2, ImageReward, and PickScore. The core claim is that simply increasing CFG scale often increases the score because it yields stronger semantic alignment, even if image quality degrades through oversaturation, artifacts, or excessive prompt locking (Xie et al., 26 Feb 2026). TDG is introduced as a deliberately revealing example of this pitfall.

The proposed corrective framework is GA-Eval, which separates effects parallel to CFG from effects orthogonal to CFG. For a non-CFG guidance method, the update is decomposed as

$\pi_\theta(a\mid s)$1

with effective guidance scale

$\pi_\theta(a\mid s)$2

The framework then compares a method not only against CFG but also against e-CFG, meaning CFG run at the matched effective guidance scale. A large positive degradation

$\pi_\theta(a\mid s)$3

is taken to indicate that apparent gains are largely attributable to increased effective guidance rather than orthogonal innovation (Xie et al., 26 Feb 2026).

The paper reports that on SD-XL and Pick-a-Pic, TDG has HPS v2 $\pi_\theta(a\mid s)$4, ImageReward $\pi_\theta(a\mid s)$5, AES $\pi_\theta(a\mid s)$6, PickScore $\pi_\theta(a\mid s)$7, and CLIPScore $\pi_\theta(a\mid s)$8 under conventional evaluation. Under GA-Eval, the corresponding winning rates for TDG on Pick-a-Pic are reported as $\pi_\theta(a\mid s)$9 for HPS v2, $$a^k = \sqrt{\bar{\alpha}_k} a^0 + \sqrt{1 - \bar{\alpha}_k}\,\epsilon,\quad \epsilon\sim\mathcal N(0,I).$$0 for ImageReward, $$a^k = \sqrt{\bar{\alpha}_k} a^0 + \sqrt{1 - \bar{\alpha}_k}\,\epsilon,\quad \epsilon\sim\mathcal N(0,I).$$1 for AES, $$a^k = \sqrt{\bar{\alpha}_k} a^0 + \sqrt{1 - \bar{\alpha}_k}\,\epsilon,\quad \epsilon\sim\mathcal N(0,I).$$2 for PickScore, and $$a^k = \sqrt{\bar{\alpha}_k} a^0 + \sqrt{1 - \bar{\alpha}_k}\,\epsilon,\quad \epsilon\sim\mathcal N(0,I).$$3 for CLIPScore, with average $$a^k = \sqrt{\bar{\alpha}_k} a^0 + \sqrt{1 - \bar{\alpha}_k}\,\epsilon,\quad \epsilon\sim\mathcal N(0,I).$$4 and $$a^k = \sqrt{\bar{\alpha}_k} a^0 + \sqrt{1 - \bar{\alpha}_k}\,\epsilon,\quad \epsilon\sim\mathcal N(0,I).$$5. On GenEval with SD-XL, TDG obtains an overall score of $$a^k = \sqrt{\bar{\alpha}_k} a^0 + \sqrt{1 - \bar{\alpha}_k}\,\epsilon,\quad \epsilon\sim\mathcal N(0,I).$$6, compared with $$a^k = \sqrt{\bar{\alpha}_k} a^0 + \sqrt{1 - \bar{\alpha}_k}\,\epsilon,\quad \epsilon\sim\mathcal N(0,I).$$7 for e-CFG and $$a^k = \sqrt{\bar{\alpha}_k} a^0 + \sqrt{1 - \bar{\alpha}_k}\,\epsilon,\quad \epsilon\sim\mathcal N(0,I).$$8 for CFG (Xie et al., 26 Feb 2026).

The broader significance of these two TDG usages lies in what they reveal about diffusion guidance as a research object. In offline RL, tied guidance motivates a modular decomposition in which guidance is trained once, frozen, and reused across policies, seeds, and algorithms. In text-to-image generation, TDG becomes an instrument for showing that guidance methods can appear to outperform CFG because evaluation itself is sensitive to large guidance scales. A plausible implication is that diffusion guidance should be analyzed not only as a sampling mechanism but also as a source of optimization and evaluation confounds.