CARINOX: Noise Optimization in Diffusion Models

• CARINOX is an inference-time framework that enhances compositional alignment by optimizing the initial noise through gradient-based refinement and best-of-N seed exploration.
• It employs a category-aware reward selection strategy by integrating multiple reward models (HPS, ImageReward, DA Score, VQA Score) to reliably match human judgments.
• The method achieves significant improvements (up to +16% on T2I-CompBench++) in handling complex compositional prompts, despite increased computational demands.

Searching arXiv for the target paper and closely related inference-time noise optimization methods. CARINOX, short for Category-Aware Reward-based Initial Noise Optimization and Exploration, is an inference-time framework for improving compositional alignment in text-to-image diffusion models by optimizing and exploring the initial noise under reward guidance, without modifying model parameters (Kasaei et al., 22 Sep 2025). It is designed for settings in which modern text-to-image systems such as Stable Diffusion, SDXL, PixArt-α, and DALL·E can generate realistic and diverse images yet often fail to satisfy prompts involving multiple objects, attributes, spatial relations, or counts. CARINOX addresses this by combining gradient-based refinement of the initial latent with best-of-$N$ seed exploration, and by selecting reward functions through a category-wise correlation analysis against human judgments rather than through a single metric or ad hoc reward combinations (Kasaei et al., 22 Sep 2025).

1. Definition and problem setting

Text-to-image diffusion models generate an image $\mathbf{x}_0$ conditioned on a text prompt $\mathbf{p}$ by starting from random noise and denoising: $$\mathbf{z}_0 \sim \mathcal{N}(0, I), \quad \mathbf{x}_0 = f_\theta(\mathbf{z}_0, \mathbf{p})$$ In this formulation, the initial noise or latent $\mathbf{z}_0$ is not merely stochastic input but a major determinant of the final image. CARINOX is built on the premise that some initializations naturally yield outputs with better compositional faithfulness than others, and that the latent can therefore be treated as an optimization variable at inference time (Kasaei et al., 22 Sep 2025).

The framework targets compositional alignment, meaning correct realization of prompts that specify multiple entities, attribute bindings, spatial arrangements, or object counts. The reported failure modes include missing entities, attribute misbinding, incorrect spatial relations, and counting errors. These failures are especially salient for prompts such as “a red car and a blue bike,” “a green glass mug with a wooden handle,” “a cat sitting under a table behind a vase,” or “five apples and two oranges” (Kasaei et al., 22 Sep 2025).

CARINOX belongs to the broader class of inference-time methods, which keep the underlying generator frozen and instead modify how generation is performed. In the source formulation, this includes prompt rewriting, attention guidance, reward-based selection, and noise or latent optimization. Within that landscape, CARINOX is specifically an instance of inference-time scaling: better alignment is obtained by spending more compute at generation time through additional seeds, optimization steps, and reward evaluations, rather than through retraining (Kasaei et al., 22 Sep 2025).

2. Antecedents and limitations of prior inference-time strategies

Prior work described in the source material falls into two broad inference-time families: noise optimization and noise exploration. Optimization-based methods such as ReNO and InitNO sample an initial noise tensor, generate an image, score it with a reward, and then update the noise by gradient ascent on that reward (Kasaei et al., 22 Sep 2025). In the notation provided,

$$\boldsymbol{\epsilon}^{(t+1)} = \boldsymbol{\epsilon}^{(t)} + \eta \, \nabla_{\boldsymbol{\epsilon}} R(G_\theta(\boldsymbol{\epsilon}^{(t)}, \mathbf{p}), \mathbf{p})$$

These methods can improve alignment, but the paper identifies three recurring limitations: sensitivity to poor initialization, unfavorable optimization trajectories in highly non-convex reward landscapes, and bias induced by optimizing a single reward such as CLIP or ImageReward (Kasaei et al., 22 Sep 2025).

Exploration-based methods such as ImageSelect, SeedSelect, SemI, and ParticleFiltering instead sample many seeds, generate an image from each, score them, and return the highest-scoring sample. This is a discrete best-of-$N$ search over the noise space. The limitations enumerated in the source are computational heaviness in high-dimensional latent spaces, lack of any refinement mechanism when no sampled seed is satisfactory, and vulnerability to incomplete or noisy rewards (Kasaei et al., 22 Sep 2025).

A central diagnostic result in the CARINOX paper is that neither a single reward metric nor naive reward combinations reliably capture the full structure of compositionality. The correlation study on T2I-CompBench++ shows that CLIPScore never appears among top metrics in any category, while different categories favor different evaluators. The source uses this to motivate a category-aware reward selection procedure and to argue that generic CLIP-based guidance can provide weak or inconsistent supervisory signal for compositional generation (Kasaei et al., 22 Sep 2025).

This suggests a broader methodological point: improvement in text-image similarity does not necessarily imply improvement in prompt-conditional structure. A plausible implication is that reward design, not only optimization strategy, is a primary bottleneck in inference-time alignment.

3. Core architecture of CARINOX

CARINOX unifies two mechanisms: continuous initial noise optimization and discrete seed exploration. The stated objective is to combine the precision of gradient-based refinement with the diversity and initialization robustness of multi-seed search, under a reward set chosen through empirical correlation with human judgments (Kasaei et al., 22 Sep 2025).

The framework has three named variants. CARINX is exploration-only, using best-of-$N$ seed selection with a fixed reward combination and no gradient optimization. CARINO is optimization-only, using multi-reward gradient ascent from a single seed. CARINOX is the full system, combining exploration and optimization (Kasaei et al., 22 Sep 2025).

The “category-aware” component is central. Compositional alignment is decomposed into categories including attributes, spatial relations, numeracy, and more complex prompt structures. On T2I-CompBench++, CARINOX evaluates candidate reward models category by category using human ratings, computes Spearman and Kendall correlations between reward scores and human judgments, and then selects reward models that consistently appear among the top three across categories (Kasaei et al., 22 Sep 2025). The selected reward set is:

• HPS
• ImageReward
• DA Score
• VQA Score

According to the source, these were chosen because they jointly cover global semantic alignment and human preference, as well as fine-grained compositional reasoning through VQA-style evaluation (Kasaei et al., 22 Sep 2025). The weights are set uniformly to $1$, and the “category-aware” designation comes from the correlation-driven selection procedure rather than from hand-tuned coefficients.

4. Mathematical formulation and optimization mechanics

In the one-step setting used by CARINOX, the generator is written as

$I = G_\theta(\boldsymbol{\epsilon}, \mathbf{p})$

with $\mathbf{x}_0$0. The source emphasizes that one-step models such as SD-Turbo, SDXL-Turbo, and PixArt-α DMD are particularly suitable because gradients from reward to noise pass through only one forward pass, thereby avoiding the vanishing or exploding gradient issues associated with multi-step diffusion backpropagation (Kasaei et al., 22 Sep 2025).

The composite reward is

$\mathbf{x}_0$1

with $\mathbf{x}_0$2 in the implementation. The optimization target is

$\mathbf{x}_0$3

The gradient with respect to the initial noise is expressed as

$\mathbf{x}_0$4

and the update follows standard gradient ascent: $\mathbf{x}_0$5

A distinctive stabilization mechanism is multi-backward gradient clipping. For each reward $\mathbf{x}_0$6, the method computes its gradient separately,

$\mathbf{x}_0$7

clips it by $\mathbf{x}_0$8 norm with threshold $\mathbf{x}_0$9,

$\mathbf{p}$0

and then aggregates the clipped reward gradients: $\mathbf{p}$1 The stated rationale is to prevent any single reward from dominating the update direction, thereby stabilizing optimization and reducing reward hacking (Kasaei et al., 22 Sep 2025).

The second stabilizer is latent regularization, which keeps the optimized latent near the Gaussian prior. Using the fact that the norm of a $\mathbf{p}$2-dimensional Gaussian vector follows a $\mathbf{p}$3 distribution, the regularizer is defined as

$\mathbf{p}$4

The final objective becomes

$\mathbf{p}$5

where $\mathbf{p}$6 controls the trade-off between reward maximization and remaining in-distribution (Kasaei et al., 22 Sep 2025).

The exploration stage samples

$\mathbf{p}$7

optimizes each seed for $\mathbf{p}$8 iterations, tracks the best per-seed image under the composite reward, and returns

$\mathbf{p}$9

The practical setting adopted from ablation is $$\mathbf{z}_0 \sim \mathcal{N}(0, I), \quad \mathbf{x}_0 = f_\theta(\mathbf{z}_0, \mathbf{p})$$0 and $$\mathbf{z}_0 \sim \mathcal{N}(0, I), \quad \mathbf{x}_0 = f_\theta(\mathbf{z}_0, \mathbf{p})$$1 (Kasaei et al., 22 Sep 2025).

5. Reward taxonomy and category-aware selection

CARINOX is evaluated against the eight compositional categories defined by T2I-CompBench++: Color, Shape, Texture, 2D Spatial, 3D Spatial, Numeracy, Non-Spatial, and Complex (Kasaei et al., 22 Sep 2025). These categories serve as the basis for the framework’s reward-selection procedure.

The paper evaluates several classes of reward metrics. The embedding-based metrics are CLIPScore, PickScore, HPS, ImageReward, and BLIP-2. The VQA-based metrics are B-VQA, DA Score, TIFA, DSG, and VQA Score. The image-only metrics are CLIP-IQA and Aesthetic Score (Kasaei et al., 22 Sep 2025). For each metric-category pair, Spearman correlation with human scores is computed, and the source further reports Kendall correlations in complementary tables.

The reward set ultimately selected for CARINOX is summarized below.

Reward	Characterization in source	Reported strengths
HPS	CLIP-based model fine-tuned on human preferences	Numeracy, 2D spatial, non-spatial relations
ImageReward	Reward model trained on human preference data for T2I	Texture, shape, overall correlation
DA Score	VQA-based; constructs questions targeting entity–attribute and relational correctness	Color, shape, texture, complex prompts
VQA Score	Evaluates yes/no questions derived from text	2D spatial, complex prompts, compositional reasoning

The source makes three comparative claims that are important for interpreting this selection. First, CLIPScore is never among top metrics in the reported category-wise analysis. Second, VQA-based scores and embedding-based preference models complement each other. Third, image-only metrics such as Aesthetic Score and CLIP-IQA poorly correlate with compositionality and are therefore excluded from the CARINOX reward set (Kasaei et al., 22 Sep 2025).

This reward composition can be written as

$$\mathbf{z}_0 \sim \mathcal{N}(0, I), \quad \mathbf{x}_0 = f_\theta(\mathbf{z}_0, \mathbf{p})$$2

Because the weights are uniform, the method’s distinctiveness lies not in coefficient tuning but in the prior empirical screening of reward candidates by category.

6. Empirical performance, scaling behavior, and resource profile

CARINOX is evaluated on T2I-CompBench++ and HRS, using SD-Turbo, SDXL-Turbo, and PixArt-α (DMD) as backbones (Kasaei et al., 22 Sep 2025). The compared methods include baseline diffusion models, attention-based guidance methods, ReNO, and exploration baselines such as Pick-A-Pic and ImageSelect.

On T2I-CompBench++, the reported mean alignment scores are as follows:

Backbone	Baseline	ReNO	CARINX	CARINO	CARINOX
SD-Turbo	0.39	0.52	0.47	0.55	0.57
SDXL-Turbo	0.41	0.51	0.50	0.56	0.57
PixArt-α DMD	0.35	0.49	0.44	0.56	0.58

The abstract reports that CARINOX raises average alignment scores by +16% on T2I-CompBench++ and consistently outperforms state-of-the-art optimization and exploration-based methods across all major categories, while preserving image quality and diversity (Kasaei et al., 22 Sep 2025).

On HRS, which covers creativity, style, object size, and visual writing, the reported mean scores are:

Backbone	Baseline	ReNO	CARINO	CARINOX
SD-Turbo	0.28	0.35	0.43	0.46
SDXL-Turbo	0.32	0.37	0.43	0.47
PixArt-α DMD	0.25	0.33	0.45	0.48

The abstract summarizes this as an average improvement of +11% on the HRS benchmark (Kasaei et al., 22 Sep 2025).

Human evaluation on T2I-CompBench++ prompts, using normalized $$\mathbf{z}_0 \sim \mathcal{N}(0, I), \quad \mathbf{x}_0 = f_\theta(\mathbf{z}_0, \mathbf{p})$$3–$$\mathbf{z}_0 \sim \mathcal{N}(0, I), \quad \mathbf{x}_0 = f_\theta(\mathbf{z}_0, \mathbf{p})$$4 ratings in $$\mathbf{z}_0 \sim \mathcal{N}(0, I), \quad \mathbf{x}_0 = f_\theta(\mathbf{z}_0, \mathbf{p})$$5, also favors the full system. For SD-Turbo, the mean rises from 0.46 at baseline to 0.62 for ReNO, 0.67 for CARINO, and 0.75 for CARINOX. For SDXL-Turbo, it rises from 0.62 to 0.69, 0.74, and 0.79, respectively. The largest gains are reported in texture, 2D spatial, color, and shape (Kasaei et al., 22 Sep 2025).

The source also reports image-quality and diversity metrics on MS-COCO for SD-Turbo variants: baseline SD-Turbo has FID 8.09, Density 0.70, Coverage 0.99; ReNO has FID 10.12, Density 0.97, Coverage 0.99; and CARINOX has FID 12.93, Density 0.91, Coverage 0.97 (Kasaei et al., 22 Sep 2025). The interpretation given is that FID worsens somewhat, but density and coverage remain high, implying only modest degradation relative to the compositional gains.

On GenEval, the paper states that CARINOX with SD-Turbo and SDXL-Turbo matches or surpasses SD3 8B and DALL·E 3 on mean scores of approximately 0.65–0.68, especially on color attribution (Kasaei et al., 22 Sep 2025).

The resource trade-off is substantial. CARINOX allocates extra inference-time compute through additional optimization steps and seed evaluations. Ablations indicate that increasing iterations helps up to about 50, after which results saturate or slightly decline, while increasing seeds helps up to about 5, after which gains also saturate (Kasaei et al., 22 Sep 2025). Memory cost rises markedly: for SD-Turbo, VRAM increases from 10 GB baseline to 15 GB ReNO and 33 GB CARINOX; the paper also notes up to ~40+ GB for SDXL-Turbo and generation times of ~60–70 s per image vs <1 s baseline (Kasaei et al., 22 Sep 2025).

7. Qualitative behavior, implications, and limitations

The qualitative analyses described in the source emphasize cases involving counts, size relations, attribute binding, style prompts, and visual writing. On prompts such as “four lamps and four dogs,” CARINOX is reported to render the specified numbers correctly where baselines miscount. On a relative-size prompt such as “a dog smaller than a chair,” it reportedly respects the intended relation where baselines often invert it. For “a striped rug under a wooden table,” the method is described as binding patterns and materials to the correct objects (Kasaei et al., 22 Sep 2025).

Iterative refinement is illustrated with the prompt “a train on the bottom of a horse.” Early iterations produce ambiguous or incorrect relations, while later iterations around 50 converge to a correctly composed scene, still showing stylistic diversity across seeds. The best-of-$$\mathbf{z}_0 \sim \mathcal{N}(0, I), \quad \mathbf{x}_0 = f_\theta(\mathbf{z}_0, \mathbf{p})$$6 selector is then used to pick the most compositionally faithful sample (Kasaei et al., 22 Sep 2025).

The source also discusses the effect of omitting Multi-Clip. Without it, strong reward-driven updates can distort realism, yielding waxy or unnatural textures even as reward scores increase. With Multi-Clip, the system is reported to better balance compositional gains with realistic appearance (Kasaei et al., 22 Sep 2025). This suggests that gradient normalization across rewards is not a peripheral heuristic but a core part of maintaining usable image manifolds during latent search.

At a broader level, the paper argues that inference-time alignment is powerful, that reward design is critical, and that noise or latent optimization provides a flexible control interface that could potentially extend to video, 3D, and editing (Kasaei et al., 22 Sep 2025). A plausible implication is that CARINOX exemplifies a more general paradigm in which generative control is offloaded from training-time parameter updates to prompt-specific search procedures over latent initializations.

The limitations are explicit. CARINOX depends on external reward models such as HPS, ImageReward, and VQA-based metrics, which may encode biases or fail to generalize to new domains. It incurs substantially greater time and memory cost than vanilla generation or ReNO. Reward mismatch remains possible, so improvements in reward may not always correspond to improvements in human judgment. Finally, very subtle compositional relations, complex logic-like prompts, and some spatial or creativity dimensions remain difficult, despite consistent gains across categories (Kasaei et al., 22 Sep 2025).

Future directions listed in the source include richer reward models trained on broader or more targeted human preference data, extension to multi-step diffusion, RL-style formulations of inference-time optimization, integration with training-time methods such as RL or DPO-style fine-tuning on reward models, and extension to temporally consistent video or 3D scene generation (Kasaei et al., 22 Sep 2025).