Align Your Tangent (AYT) Overview

• AYT is a training objective for consistency models that aligns output tangents with the data manifold using a self-supervised feature-space distance.
• It replaces the traditional pixel-space loss with a manifold feature distance, enhancing convergence speed and robustness even with small batch sizes.
• Empirical results on CIFAR-10 and ImageNet 64×64 show improved one-step FID scores and faster training compared to conventional CM objectives.

Align Your Tangent (AYT) is a training objective for consistency models (CMs) that seeks to make CM output update directions, termed “tangents,” point toward the data manifold rather than move parallel to it. Introduced in the context of accelerating one- and two-step generative sampling, AYT replaces the usual pixel-space consistency distance with a self-supervised feature-space distance, the manifold feature distance (MFD), whose geometry is intended to induce manifold-aligned tangents during training. In the reported experiments, the method is built on top of Easy Consistency Training (ECT), uses a VGG16-based auxiliary feature network trained from scratch, and is evaluated on CIFAR-10 and ImageNet $64\times 64$ with improvements in convergence speed, one-step FID, and robustness to small batch sizes (Kim et al., 1 Oct 2025).

1. Concept and problem formulation

AYT was proposed for the training of consistency models, a class of generative models trained to be consistent on diffusion or probability flow ordinary differential equation (PF-ODE) trajectories. The motivating observation is that, although CMs enable one- or two-step sampling, they “typically require prolonged training with large batch sizes to obtain competitive sample quality” (Kim et al., 1 Oct 2025).

The paper formalizes standard CM training in discrete time as

$\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\Big[(\Delta t)^{-1} d\big(f_\theta(x_t,t), f_{\text{sg}(x_{t-\Delta t}, t-\Delta t)\big)\Big], \tag{1}$

where $d$ may be MSE, pseudo-Huber, or LPIPS, and $f_{\text{sg}}$ denotes the stop-gradient target (Kim et al., 1 Oct 2025). With squared Euclidean distance, the same objective is rewritten in the gradient form

$\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\left[f_\theta(x_t,t)^\top \left(\Delta f_{\text{sg}(x_t,t)/\Delta t\right)\right], \tag{2}$

with

$\Delta f_{\text{sg}(x_t,t)/\Delta t \coloneqq \frac{f_{\text{sg}(x_t,t)-f_{\text{sg}(x_{t-\Delta t},t-\Delta t)}{\Delta t}. \tag{3}$

Taking $\Delta t \to 0$ yields the continuous objective

$\min_{\theta}\; \mathbb{E}_{x,t}\left[f_\theta(x_t,t)^\top \left(df_{\text{sg}(x_t,t)/dt\right)\right]. \tag{4}$

Within this formulation, AYT focuses on the geometry of the tangent signal. The method does not introduce a new generative architecture; rather, it changes the training objective so that the optimization geometry is defined in a feature space designed to encode off-manifold perturbations (Kim et al., 1 Oct 2025).

2. Oscillatory tangents and the manifold-alignment hypothesis

The central diagnosis behind AYT is that CM tangents are “quite oscillatory” near convergence, in the sense that they move parallel to the data manifold rather than toward it (Kim et al., 1 Oct 2025). The paper defines the tangent as the instantaneous derivative of the CM output along the PF-ODE path: $\frac{d f_{\text{sg}(x_t,t)}{dt} = \nabla_{x_t} f_{\text{sg}(x_t,t)\left(\frac{d x_t}{dt}\right) + \frac{\partial f_{\text{sg}(x_t,t)}{\partial t}. \tag{5}$

The authors’ empirical interpretation is that these tangents contain “non-trivial components parallel to the data manifold,” may move noisy inputs “sideways” along the manifold, and can change substantially under small perturbations in training (Kim et al., 1 Oct 2025). They argue that this matters because CM training aims to contract paths from noise to data; if the tangent predominantly follows the manifold instead of approaching it, convergence is slowed and gradient variance is plausibly increased. This suggests a geometric failure mode in which the learned local update field is not aligned with the normal directions that would most directly denoise samples.

The paper reports this behavior on CIFAR-10 and on a 2D synthetic disc dataset. On the latter, the authors explicitly decompose tangents into manifold-parallel and manifold-orthogonal components and find large parallel components in vanilla CMs (Kim et al., 1 Oct 2025). A plausible implication is that the sample-quality plateau observed under standard objectives can coexist with inefficient local dynamics in the tangent field.

A related but distinct tangent-alignment idea appears in interpretability research, where tangent-space alignment is used as a criterion for “user-friendly” explanations on data manifolds. In that setting, tangential alignment is formalized via a tangent-space projection fraction $\mu_x(v)$, and the baseline of Integrated Gradients is optimized so that the explanation lies in the tangent space at the input (Simpson et al., 11 Mar 2025). The two works address different tasks, but both organize their objectives around the geometry of tangent directions relative to an underlying manifold.

3. Manifold feature distance and learned manifold features

AYT addresses oscillatory tangents by replacing pixel-space distance with a feature-space distance

$\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\Big[(\Delta t)^{-1} d\big(f_\theta(x_t,t), f_{\text{sg}(x_{t-\Delta t}, t-\Delta t)\big)\Big], \tag{1}$0

where $\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\Big[(\Delta t)^{-1} d\big(f_\theta(x_t,t), f_{\text{sg}(x_{t-\Delta t}, t-\Delta t)\big)\Big], \tag{1}$1 is a learned feature map (Kim et al., 1 Oct 2025). With squared feature distance, the paper gives the tangent-form objective

$\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\Big[(\Delta t)^{-1} d\big(f_\theta(x_t,t), f_{\text{sg}(x_{t-\Delta t}, t-\Delta t)\big)\Big], \tag{1}$2

and writes the feature-space tangent as

$\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\Big[(\Delta t)^{-1} d\big(f_\theta(x_t,t), f_{\text{sg}(x_{t-\Delta t}, t-\Delta t)\big)\Big], \tag{1}$3

This representation is the basis of the manifold feature distance. Because the tangent becomes a linear combination of feature gradients $\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\Big[(\Delta t)^{-1} d\big(f_\theta(x_t,t), f_{\text{sg}(x_{t-\Delta t}, t-\Delta t)\big)\Big], \tag{1}$4, the training geometry depends on how the features are constructed. The paper’s design principle is to choose $\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\Big[(\Delta t)^{-1} d\big(f_\theta(x_t,t), f_{\text{sg}(x_{t-\Delta t}, t-\Delta t)\big)\Big], \tag{1}$5 so that $\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\Big[(\Delta t)^{-1} d\big(f_\theta(x_t,t), f_{\text{sg}(x_{t-\Delta t}, t-\Delta t)\big)\Big], \tag{1}$6 on the clean data manifold $\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\Big[(\Delta t)^{-1} d\big(f_\theta(x_t,t), f_{\text{sg}(x_{t-\Delta t}, t-\Delta t)\big)\Big], \tag{1}$7, while level sets $\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\Big[(\Delta t)^{-1} d\big(f_\theta(x_t,t), f_{\text{sg}(x_{t-\Delta t}, t-\Delta t)\big)\Big], \tag{1}$8 correspond to increasingly perturbed versions of $\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\Big[(\Delta t)^{-1} d\big(f_\theta(x_t,t), f_{\text{sg}(x_{t-\Delta t}, t-\Delta t)\big)\Big], \tag{1}$9 for larger $d$0 (Kim et al., 1 Oct 2025). Since gradients of scalar functions are orthogonal to level sets, $d$1 then tends to point toward directions that reduce the corresponding perturbation.

The learned coordinates are called manifold features. They are trained from perturbation families $d$2 using the self-supervised regression objective

$d$3

The transformation families reported in the paper comprise 15 total dimensions:

• Degradations: Gaussian noise, Gaussian blur, Mixup.
• Geometric: isotropic scaling, anisotropic scaling, fractional rotation, fractional translation.
• Color: brightness, contrast, hue, saturation.

For Gaussian blur, the training objective is written as

$d$4

where $d$5 is Gaussian blur with standard deviation $d$6 (Kim et al., 1 Oct 2025).

The paper characterizes this feature construction as self-supervised and manifold-sensitive. This suggests that AYT’s main novelty is not the generic use of feature-space loss, but the use of features whose coordinates are explicitly trained to quantify off-manifold perturbation strength.

4. Training procedure and implementation context

AYT consists of two stages. First, auxiliary feature maps $d$7 are trained to predict transformation strengths $d$8 from transformed images using Eq. (8). Second, CM training replaces the usual pixel-space consistency loss with the feature-space loss

$d$9

Conceptually, this substitutes the feature-space tangent of Eq. (7) for the unconstrained pixel-space tangent (Kim et al., 1 Oct 2025).

The paper emphasizes that, under the original CM objective, the effective feature map is $f_{\text{sg}}$0, so the Jacobian is the identity and the tangent can point in arbitrary directions. AYT changes this geometry by imposing a feature representation trained to encode degradations, geometric transforms, and color shifts (Kim et al., 1 Oct 2025).

The reported implementation context is specific. AYT is built on top of Easy Consistency Training (ECT). The classifier / feature network is VGG16, trained from scratch. The CM backbones are DDPM++ for CIFAR-10 and EDM2-S for ImageNet $f_{\text{sg}}$1 (Kim et al., 1 Oct 2025).

The training details recorded in the paper are as follows:

Component	Setting
Feature training on CIFAR-10	Adam, lr $f_{\text{sg}}$2, batch 512, 400K iterations
CM training	RAdam or Adam depending on dataset setup
CIFAR-10 CM training	400K iterations, no multi-stage schedule
ImageNet $f_{\text{sg}}$3 CM training	200K iterations; AYT enabled after 75K iterations to reduce early overfitting
Default batch size	128
Batch-size ablation	16, 32, 64, 128
Evaluation	FID on 50K generated samples; 1-step and 2-step sampling; fixed intermediate timestep for 2-step sampling

The paper also notes that the method requires training an auxiliary feature network and therefore adds computation and memory overhead, though it is described as lightweight relative to the main model (Kim et al., 1 Oct 2025).

5. Empirical results and ablations

The main empirical claim is that AYT can accelerate CM training by orders of magnitude compared with pseudo-Huber-based CM training (Kim et al., 1 Oct 2025). The paper further states that AYT can even outperform LPIPS-based training and that its features, unlike LPIPS, are trained on the target dataset itself rather than inherited from ImageNet pretraining.

On CIFAR-10, the paper reports:

• ECT: 1-step FID $f_{\text{sg}}$4, 2-step FID $f_{\text{sg}}$5
• ECT + AYT: 1-step FID $f_{\text{sg}}$6, 2-step FID $f_{\text{sg}}$7

On ImageNet $f_{\text{sg}}$8, it reports:

• ECT-S: 1-step FID $f_{\text{sg}}$9, 2-step FID $\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\left[f_\theta(x_t,t)^\top \left(\Delta f_{\text{sg}(x_t,t)/\Delta t\right)\right], \tag{2}$0
• ECT-S + AYT: 1-step FID $\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\left[f_\theta(x_t,t)^\top \left(\Delta f_{\text{sg}(x_t,t)/\Delta t\right)\right], \tag{2}$1, 2-step FID $\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\left[f_\theta(x_t,t)^\top \left(\Delta f_{\text{sg}(x_t,t)/\Delta t\right)\right], \tag{2}$2

These results are presented as substantial gains in one-step quality while maintaining competitive two-step performance (Kim et al., 1 Oct 2025). The paper also states that AYT remains competitive with or better than some distillation-based methods, despite being trained from scratch and not relying on a pretrained teacher.

The LPIPS comparison is singled out as particularly significant. The paper reports that LPIPS can exhibit degraded two-step quality after extended training and that denoising-FID at an intermediate noise level $\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\left[f_\theta(x_t,t)^\top \left(\Delta f_{\text{sg}(x_t,t)/\Delta t\right)\right], \tag{2}$3 diverges for LPIPS-trained CMs. The authors speculate that this may arise from mismatch between ImageNet-trained LPIPS features and the CIFAR-10 data distribution, or from inaccurate small-$\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\left[f_\theta(x_t,t)^\top \left(\Delta f_{\text{sg}(x_t,t)/\Delta t\right)\right], \tag{2}$4 outputs corrupting larger-$\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\left[f_\theta(x_t,t)^\top \left(\Delta f_{\text{sg}(x_t,t)/\Delta t\right)\right], \tag{2}$5 outputs (Kim et al., 1 Oct 2025). AYT is positioned as avoiding this mismatch by learning features on the target dataset.

The small-batch ablation is another major result. The paper states that AYT achieves competitive FID with batch size as small as 16, and in ablations can outperform ECT trained with batch size 128 (Kim et al., 1 Oct 2025). This is presented as evidence consistent with the oscillatory-tangent hypothesis: reducing manifold-parallel tangent components appears to reduce variance and stabilize optimization.

The transformation-group ablations compare:

• DEG: degradations
• GEO: geometric transforms
• CLR: color transforms

The reported finding is that combining more transformations consistently helps, and that geometric transformations give the biggest improvement (Kim et al., 1 Oct 2025). The paper interprets this as indicating that vanilla CM tangents are particularly weak in geometric or off-manifold directions.

6. Interpretation, scope, and limitations

AYT is framed as a change in loss geometry. Instead of optimizing consistency in pixel space, it optimizes consistency in a feature space whose gradients are intended to point toward the data manifold (Kim et al., 1 Oct 2025). In this formulation, tangent alignment is not an auxiliary diagnostic but the central mechanism by which training is accelerated.

The method is explicitly limited by the scope of the reported experiments. The paper notes that experiments are restricted to images and resolutions up to $\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\left[f_\theta(x_t,t)^\top \left(\Delta f_{\text{sg}(x_t,t)/\Delta t\right)\right], \tag{2}$6, and that broader applicability to audio, text, or higher-resolution latent pipelines is suggested but not demonstrated (Kim et al., 1 Oct 2025). The reliance on an auxiliary feature network is also a structural limitation, even if the authors characterize the overhead as modest relative to the CM itself.

A broader conceptual context can be drawn from other tangent-centered formulations. In explainability, tangential alignment has been used to choose Integrated Gradients baselines so that attributions lie in the tangent space of a data manifold (Simpson et al., 11 Mar 2025). In geometric analysis, “curve shortening” has been reinterpreted as “tangent aligning” through the direction energy

$\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\left[f_\theta(x_t,t)^\top \left(\Delta f_{\text{sg}(x_t,t)/\Delta t\right)\right], \tag{2}$7

with the gradient flow remaining the curve shortening flow (Miura et al., 4 Apr 2025). These works do not address consistency models, but they indicate that tangent alignment functions more generally as a geometric organizing principle across disparate domains.

Within generative modeling, however, AYT refers specifically to the CM objective based on manifold feature distance and manifold-aligned tangents (Kim et al., 1 Oct 2025). Its significance lies in shifting attention from output-level consistency alone to the directionality of the local update field that training induces.

7. Position within consistency-model research

AYT is situated within work on reducing inference time in diffusion and flow-matching systems without sacrificing sample quality. Consistency models are attractive because they enable one- or two-step flow or diffusion sampling, but the paper argues that their training dynamics near convergence are a key bottleneck (Kim et al., 1 Oct 2025). In this framing, AYT addresses not sampling mechanics directly but the optimization inefficiency of conventional CM objectives.

The method’s reported advantages can be summarized in three technical terms. First, it alters the tangent signal by replacing $\min_{\theta}\; \mathbb{E}_{x,t,\Delta t}\left[f_\theta(x_t,t)^\top \left(\Delta f_{\text{sg}(x_t,t)/\Delta t\right)\right], \tag{2}$8 with a learned, self-supervised manifold feature map. Second, it appears to suppress manifold-parallel oscillatory components in the tangent field, as observed on CIFAR-10 and synthetic disc data. Third, it improves training efficiency and small-batch robustness while remaining compatible with standard CM backbones and ECT-style training pipelines (Kim et al., 1 Oct 2025).

The paper’s stated bottom line is that AYT improves consistency model training by making tangents manifold-aligned through a self-supervised manifold feature space, thereby reducing oscillatory behavior, improving sample quality, speeding convergence dramatically, and making training more robust at small batch sizes (Kim et al., 1 Oct 2025). This suggests a shift in how CM objectives may be designed: from purely matching outputs across timesteps to engineering the geometry of the tangent directions that the objective induces.