Not All Prediction Targets Keep Training-Free Diffusion Guidance on the Manifold

Abstract: Training-free guidance (TFG) steers a pretrained diffusion model toward a desired attribute at inference. To be effective, this guidance must be applied from the earliest, high-noise steps of sampling. Because its objective (a classifier or energy) is defined on clean images, $ε$- and $v$-prediction models must first estimate the clean image $\hat{x}$ from the noisy state at each step, and the accuracy of that estimate determines how easily guidance drifts off the data manifold. $x$-prediction, a recent alternative, outputs the clean image directly, removing this source of error even at high noise. This is our motivation. We provide a theoretical analysis of how each prediction target shapes this accuracy, and introduce guided-class FID (Child FID), a metric that exposes the manifold damage standard evaluation misses. Experiments on a new fine-grained bird benchmark and on style transfer confirm that $x$-prediction keeps guided samples on the manifold most reliably, making it the strongest foundation for training-free guidance. Code is available at https://github.com/ManLuML/on-manifold-tfg

• The paper shows that x-prediction prevents error amplification, ensuring on-manifold consistency and superior performance in diffusion guidance.
• It employs rigorous theoretical analysis and empirical evaluations using synthetic datasets and an ImageNet-based benchmark to compare prediction targets.
• Findings recommend x-prediction for robust, high-fidelity generation under strong guidance, directly influencing model design and control.

Training-Free Diffusion Guidance and the Crucial Role of Prediction Targets

Overview

This paper investigates the manifold preservation properties of training-free guidance (TFG) in diffusion models, focusing on how the choice of prediction target—specifically, $\epsilon$ (noise), $v$ (velocity), or $x$ (clean data)—fundamentally determines the fidelity of generated samples under inference-time guidance. The authors provide a rigorous theoretical hierarchy of error amplification, empirically expose differences via a new fine-grained benchmark and manifold-aware metrics, and demonstrate that $x$-prediction is uniquely robust, ensuring guided samples remain on the data manifold across guidance strengths, dimensional regimes, and various tasks.

Figure 1: Guidance schematic illustrating how prediction target determines whether TFG stays on the data manifold (left: schematic, right: visual outcomes of crossed-lines and ImageNet samples; $x$-prediction preserves manifold structure while $\epsilon$-prediction collapses off-manifold).

Theoretical Analysis: Error Amplification Hierarchy

The central theoretical contribution is a precise quantification of how each prediction target amplifies network prediction error in the estimation of $x$, the clean image, which is essential for gradient-based guidance. The recovery formulas induce distinct error amplification:

• $\epsilon$-prediction: Clean image estimate error is amplified by $(1-t)/t$, diverging as $t\rightarrow 0$ (early, noisy steps), causing uncontrolled perturbations off the data manifold.
• $v$0-prediction: Error is attenuated by a bounded $v$1 factor.
• $v$2-prediction: No error amplification; the model outputs clean $v$3 directly.

Cumulative analysis across denoising steps proves that $v$4-prediction's errors compound and diverge, $v$5-prediction's stay bounded, and $v$6-prediction’s remain the lowest.

Empirical Evidence: Controlled and Large-Scale Evaluation

The paper systematically contrasts these prediction targets both through toy distributions and ImageNet-scale experiments:

• Synthetic Controlled Setting: On the crossed-lines dataset embedded in up to 512 dimensions, all architectural and training factors are matched except for the prediction target. $v$7-prediction retains $v$8 on-manifold rate at $v$9; $x$0-prediction is $x$1; $x$2-prediction collapses to $x$3.
• ImageNet Birds Benchmark: Introduction of a fine-grained (143 species) ImageNet-based benchmark, using a strict separation of guidance and evaluation classifiers to prevent circularity.
• Child FID (C-FID) Metric: A domain-specific, manifold-sensitive extension of FID (Fréchet Inception Distance), computed between guided samples and the reference set for the target class instead of the full ImageNet marginal.

Figure 3: P-FID vs. Validity tradeoff on birds: Increasing Validity does not guarantee low distributional divergence from the parent class reference, masking off-manifold collapse visible in C-FID.

Key empirical finding is that at matched classifier accuracy (Validity $x$4), $x$5-prediction delivers C-FID 32.9, whereas $x$6-prediction achieves only 38.1 (higher is worse, indicating manifold damage), despite producing similar or greater Validity—i.e., guided samples fool the classifier but are not on-manifold. This effect generalizes across capacities, tasks (classification, style transfer, inverse problems), and architectures.

Guidance-Strength Pareto Frontiers and Diversity-Quality Tradeoff

The authors propose not to report single operating points but instead plot full Pareto frontiers (P-FID vs. Validity, P-FID vs. C-FID) across guidance strengths $x$7 for each model. Under strong guidance, $x$8- and $x$9-prediction models collapse to narrow, classifier-fooling sets (high precision, low recall), while $x$0-prediction maintains higher diversity and better manifold coverage.

Figure 2: Precision vs. Recall curves on fine-grained bird generation as guidance strength increases: $x$1-prediction achieves artificially high precision (fidelity) via mode collapse, while $x$2-prediction alone achieves higher recall (diversity), robustly covering the target class.

Qualitative inspections show that only $x$3-prediction maintains compositionally diverse and photorealistic outputs at high guidance strengths; $x$4-prediction fails with significant mode concentration and loss of realism.

Implications for Model and System Design

The findings directly affect best practices for inference-time control:

• Manifold preservation under strong guidance is not a property of the guidance algorithm per se but of the underlying prediction target.
• $x$5-prediction yields not only higher on-manifold rates but allows much stronger guidance strengths before degradation, directly enabling more aggressive and precise control without distributional collapse.
• The theoretical hierarchy predicts that the gap between prediction targets widens with data dimensionality—a prediction confirmed empirically.
• Robust guidance enables higher reliability for downstream tasks, e.g., conditional generation, style transfer, and inverse problems such as deblurring and super-resolution.

Practical Recommendations and Limitations

For practitioners and model designers targeting training-free, plug-and-play conditional generation or editing with diffusion models, the results imply:

• Opt for $x$6-prediction as the model target whenever possible, especially in high-dimensional (e.g., pixel) spaces.
• Evaluate guided models not just with classifier Validity but with task- and class-specific manifold-aware metrics (e.g., C-FID, PR).
• Guidance evaluation must involve full strength sweeps and Pareto analysis, not fixed points, to expose robustness limits.

The authors note that while the primary finding is robust across multiple architectures and two data spaces (pixel and latent), no single real-world comparison can match $x$7-, $x$8-, and $x$9-prediction under identical architectures and operating spaces at modern ImageNet scale due to architectural support constraints.

Future Directions

Future research directions highlighted include:

• Application to inference-time scaling algorithms and Monte Carlo-based search/steering (e.g., SMC, guide-your-own, test-time alignment), where the prediction target's error properties will govern sample quality and efficiency.
• Extension to long-horizon or spatiotemporal domains (e.g., video, text-to-image) where error amplification compound across frames could further magnify the observed hierarchy.
• Investigation into the role of manifold curvature, intrinsic dimension, and guidance objective class (beyond classification and style transfer).

Conclusion

The choice of prediction target is a first-order concern for guided diffusion inference. $x$0-prediction is uniquely suited to preserving the core generative modeling contract—plausible, on-manifold samples—even under strong, training-free guidance. This result changes the recommended default for both researchers and practitioners designing controllable, guidance-robust diffusion models.

Figure 6: Visualization of crossed-lines guided generation at high dimensionality: $x$1-prediction alone maintains structural integrity with increased ambient dimension.

• Not All Prediction Targets Keep Training-Free Diffusion Guidance on the Manifold (2607.00647)
• Li & He, "Back to Basics: Let Denoising Generative Models Denoise" (Li et al., 17 Nov 2025)
• Ye et al., "TFG: Unified Training-Free Guidance for Diffusion Models" (NeurIPS) [2024]
• Peebles & Xie, "Scalable Diffusion Models with Transformers" (Peebles et al., 2022)