Null-TTA: Test-Time Alignment in Diffusion Models

• Null-TTA is a paradigm that adapts text-to-image diffusion models at test time by optimizing the null-text embedding for controlled semantic output.
• It leverages classifier-free guidance and the pretrained semantic manifold to enhance reward alignment while mitigating issues like under-optimization and reward hacking.
• The method achieves state-of-the-art performance with improved reward metrics and visual fidelity without updating model parameters.

Null-Text Test-Time Alignment (Null-TTA) is a paradigm for the test-time adaptation of text-to-image diffusion models. It aligns a pre-trained diffusion model’s output distribution towards a user-specified reward objective during inference by optimising the “null-text” embedding─the anchor vector for unconditional generation in classifier-free guidance─while not updating model parameters. By operating in the structured, semantically coherent manifold defined by the output of a pretrained text encoder (e.g., CLIP), Null-TTA aims to circumvent issues of under-optimisation and reward hacking, achieving state-of-the-art alignment of generated samples with target rewards while maintaining generalisation across secondary metrics (Kim et al., 25 Nov 2025).

1. Background: Test-Time Alignment and Classifier-Free Guidance

Test-Time Alignment (TTA) refers to methods that adapt a pre-trained generative model at inference to maximise a user-defined reward function $R(x)$, without parameter updates. TTA methods seek to shift the output distribution $p(x)$ such that $\mathbb{E}_{x\sim p}[R(x)]$ is maximised. Under-optimisation arises when the method yields only marginal improvements, whereas reward hacking refers to over-optimisation that exploits pathologies in the latent or noise space (e.g., injecting noise artifacts) to spuriously increase $R(x)$, at the cost of visual fidelity, diversity, or other metrics.

Classifier-Free Guidance (CFG) is prevalent in text-to-image diffusion models such as Stable Diffusion. CFG modifies the predicted noise at each sampling step through the formula:

$$\tilde \varepsilon_\theta(x_t, t, c, \varphi) = \varepsilon_\theta(x_t, t, \varphi) + s \cdot [\varepsilon_\theta(x_t, t, c) - \varepsilon_\theta(x_t, t, \varphi)]$$

where $c$ is the conditional text embedding, $\varphi$ is the “null-text” embedding produced by encoding the empty prompt, and $s > 1$ is the guidance scale. The null-text embedding $\varphi$ serves as the baseline anchor for the model’s generative distribution; modifications to $\varphi$ shift the manifold of generated outputs in a semantically meaningful manner, unlike arbitrary manipulations of noise or latent variables.

2. Null-TTA Methodology: Optimisation in Semantic Space

Null-TTA reframes TTA as the optimisation of the null-text embedding $\varphi'$ to directly steer the generative distribution $p(x;\varphi')$ towards maximising the target reward, with regularisation to constrain deviation from the pre-trained generative prior. The target is to maximise $\mathbb{E}_{x_0\sim p(x_0|\varphi')}[R(x_0)]$, subject to a KL penalty:

$$\max_{\varphi'} \left\{\lambda_1 \mathbb{E}_{x_0\sim p(x_0|\varphi')}[R(x_0)] - \lambda_2 \mathrm{KL}\left(p(x_{0:T};\varphi') \Vert p(x_{0:T};\varphi)\right)\right\}$$

Expanding this per step under Gaussian assumptions for $\varphi, \varphi'$, the per-time-step objective (Eq. 13) becomes:

$$\max_{\varphi'} \Biggl[ \lambda_1 R(\hat{x}_0(x_t,\varphi')) - \lambda_2 \sum_{i=1}^T w_i \|\tilde\varepsilon(x_i,\varphi') - \tilde\varepsilon(x_i,\varphi)\|^2 - \frac{\lambda_2}{2\sigma_\varphi^2} \|\varphi' - \varphi\|^2 \Biggr]$$

where $\hat{x}_0(x_t,\varphi')$ is Tweedie’s posterior mean estimate of the clean sample; $w_i$ are schedule-derived weights. The loss can equivalently be written as a penalised objective: $$L(\varphi') = -R(\hat{x}_0(x_t,\varphi')) + \mu \|\varphi' - \varphi\|^2$$ with a consistency term on noise prediction.

Gradients $\partial L/\partial \varphi'$ can be obtained by backpropagating through the U-Net’s cross-attention layers or using finite-difference estimators for non-differentiable rewards. Adam is used for optimisation, typically with learning rate $\approx 10^{-2}$. Regularisation strength ($\lambda_2$) is usually annealed, starting large to prevent drift early in denoising when the reward proxy is noisy.

3. Inference-Time Procedure and Algorithmic Design

The test-time algorithm for Null-TTA proceeds as follows:

• Initialise latent $z_T \sim \mathcal{N}(0,I)$, $\varphi \gets$ TextEncoder(""), $c \gets$ TextEncoder(prompt), $\varphi' \gets \varphi$.
• For each reverse diffusion timestep $t$:
  • Calculate schedule parameters $\alpha_t, \bar{\alpha}_t$ and step-specific regularisation.
  • Perform $N_t$ inner optimisations of $\varphi'$ to maximise the per-step objective using the current reward $R(\hat{x}_0)$ and regularisation terms.
  • Employ particle filtering: for $K$ samples, propagate $$z_{t-1}^{(k)} \sim \mathcal{N}(\mu_t(z_t,\varphi'), \sigma_t^2 I)$$, select the sample with the highest reward proxy.
• Output the final decoded image from $z_0$.

Key hyperparameters: base model Stable Diffusion v1.5, $T=100$ steps, guidance $s=7.5$, $N_{\text{min}}=5$, $N_{\text{max}}$ in $\{25,\ldots,115\}$, $K=3$, Adam optimiser. Compute overhead with $N_{\text{max}}=55$ is approximately 8m40s/image on NVIDIA L40S (17.6 GB), which is lower than DNO (19m38s, 20.4 GB).

4. Semantic Manifold Structure and Prevention of Reward Hacking

Unlike optimisation in noise or latent space, Null-TTA operates strictly on the space of null-text embeddings derived from a pretrained semantic encoder (e.g., CLIP). This space exhibits a smooth, coherent manifold: modifications to $\varphi'$ correspond to controlled, prompt-like semantic changes in generated images (e.g., altering color, style, or compositional properties) rather than arbitrary texture noise. This property prevents reward hacking, as non-semantic axes—responsible for visual artifacts—cannot be accessed without incurring destructive KL penalties.

Empirical evidence from Table 1 shows that Null-TTA maintains or improves held-out rewards (e.g., PickScore, ImageReward) compared to latent/noise-based TTA, which degrade under aggressive reward maximisation. Prior work on Null-Text Inversion by Mokady et al. corroborates the preservation of fine semantics and fidelity when optimising null-text embeddings.

5. Comparative Experimental Evaluation

Quantitative evaluation demonstrates that Null-TTA expands the Pareto frontier across various target rewards and step budgets. In Table 1 (PickScore as target), Null-TTA achieves the highest or competitive values across all reward metrics:

Method	PickScore ↑	HPSv2 ↑	Aesthetic ↑	ImageReward ↑
SD-v1.5	0.218	0.279	5.232	0.339
DNO	0.289	0.290	5.075	0.396
DAS	0.258	0.289	5.382	0.871
Null-TTA	0.315	0.294	5.431	0.946

When optimising multi-objective trade-offs (e.g., weighted averages of PickScore and HPSv2), Null-TTA results in superior trade-off curves: for any trade-off parameter $w$, secondary metric degradation is less than DAS. Qualitative analysis also favours Null-TTA, especially for prompts demanding compositional reasoning, counting, unusual coloring, and spatial arrangements, where prompt adherence is higher than with baselines.

6. Implementation Considerations and Limitations

Empirically robust settings for rewards and regularisation were established: for HPSv2 and PickScore, $$\lambda_1=100, \lambda_2=0.002, \sigma_\varphi^2=0.01$$; for Aesthetic, $$\lambda_1=2, \lambda_2=0.002, \sigma_\varphi^2=0.01$$. Annealing parameter $\gamma=0.008$ co-controls both regularisation decay and growth of inner optimisation steps. The method achieves competitive compute performance and is efficient without model fine-tuning.

Limitations include:

• Early denoising steps rely on Tweedie estimates of $\hat{x}_0$, which can be unreliable; thus, strong initial regularisation is required.
• Direct gradient-based optimisation requires differentiable $R(x)$ or incurs higher computational cost with finite-difference estimation.
• Null-TTA has only been demonstrated with text-to-image diffusion; extension to other modalities depends on verifying their conditioning spaces also admit a semantically coherent manifold.

Potential future extensions include learned reward proxies for early denoising steps, automated multi-objective scheduling, adaptation to non-differentiable scientific simulators, and joint optimisation of $\varphi'$ with a subset of U-Net parameters.

7. Significance and Outlook

Null-Text Test-Time Alignment establishes semantic-space optimisation as a principled alternative to latent/noise-based TTA for diffusion models (Kim et al., 25 Nov 2025). By harnessing the structure and interpretability of pretrained text embedding manifolds, Null-TTA achieves alignment of generative distributions with user-specified objectives while preserving cross-reward generalisation, visual quality, and diversity. This approach overcomes both the under- and over-optimisation drawbacks that beset prior TTA techniques, and delineates a promising research direction for robust, semantically faithful inference-time adaptation in generative models.