Diffusion Mental Averages

Abstract: Can a diffusion model produce its own "mental average" of a concept-one that is as sharp and realistic as a typical sample? We introduce Diffusion Mental Averages (DMA), a model-centric answer to this question. While prior methods aim to average image collections, they produce blurry results when applied to diffusion samples from the same prompt. These data-centric techniques operate outside the model, ignoring the generative process. In contrast, DMA averages within the diffusion model's semantic space, as discovered by recent studies. Since this space evolves across timesteps and lacks a direct decoder, we cast averaging as trajectory alignment: optimize multiple noise latents so their denoising trajectories progressively converge toward shared coarse-to-fine semantics, yielding a single sharp prototype. We extend our approach to multimodal concepts (e.g., dogs with many breeds) by clustering samples in semantically-rich spaces such as CLIP and applying Textual Inversion or LoRA to bridge CLIP clusters into diffusion space. This is, to our knowledge, the first approach that delivers consistent, realistic averages, even for abstract concepts, serving as a concrete visual summary and a lens into model biases and concept representation.

• The paper presents a novel method that aligns denoising trajectories to produce high-fidelity, semantically meaningful prototype images.
• It utilizes iterative optimization in the U-Net h-space and integrates mode discovery to handle both unimodal and multimodal concept representations.
• Empirical results demonstrate DMA’s superior representativeness, minimal intra-prototype variance, and its capability to reveal latent model biases.

Diffusion Mental Averages: Model-Centric Concept Prototyping via Trajectory Alignment

Motivation and Background

The paper "Diffusion Mental Averages" (2603.29239) introduces a model-centric methodology for generating semantically meaningful prototype images—termed "mental averages"—directly from pre-trained diffusion models. Standard approaches such as pixel-space averaging or VAE/GAN latent space averaging either yield visually unrealistic or semantically ambiguous prototypes when applied to diffusion models. This is mainly because diffusion models lack an explicit, decodable, semantic latent "bottleneck" layer, and their semantic representations are distributed across the denoising trajectory. The significance of this technique is multifold: it creates consistent, sharp, and realistic visual prototypes, exposes model biases and internal conceptual organization, and provides interpretability tools absent in previous diffusion model analyses.

Figure 1: Overview of the DMA pipeline: Multiple noise latents are jointly optimized so their denoising trajectories converge toward shared semantics via progressive h-space mean alignment, yielding a single mental average for the concept.

Methodology: Diffusion Mental Averaging

DMA redefines concept prototyping as a trajectory alignment task in diffusion models. Rather than averaging generated samples or aligning latents only within a static layer, DMA optimizes multiple noise latents so that their denoising trajectories converge toward a continually updated consensus at a semantically meaningful U-Net layer, referred to as $h$-space. Through iterative optimization at each diffusion timestep and synchronizing h-space feature means, DMA enables a coarse-to-fine semantic alignment: global structure is stabilized early, while lower-level details accrue throughout the denoising process.

Algorithmically, for a set of $K$ noise vectors, at each timestep $t$ (up to a cutoff $t_\text{stop}$), the h-space mean across all optimized latents is computed and used as the target for further optimization. Subsequently, DDIM is used for denoising to the next timestep, and the process iterates. The final prototype is decoded via the VAE decoder from any optimized latent. This mechanism ensures that the alignment occurs in the intrinsic semantic space, progresses temporally with the model’s generative hierarchy, and does not require architectural changes or additional supervision.

Mode Discovery and Conditioning

While basic DMA yields a single prototype per prompt, many real-world concepts are visually multimodal. To address this, the authors integrate a two-stage mode discovery pipeline. First, samples are embedded and clustered in a stable, high-level semantic space (CLIP or BLIP-VQA), supporting both unsupervised and attribute-grounded clustering. Each cluster is then distilled using DMA, augmented with either Textual Inversion or LoRA for cluster-conditioned guidance, mitigating the semantic space mismatch between clustering (e.g., CLIP) and U-Net h-space.

Figure 2: DMA prototypes on unsupervised modes for concepts with both single and multimodal structure, illustrating improved disambiguation relative to a naïve mean.

Figure 3: DMA prototypes for grounded modes by relevant attributes (e.g., breed, ethnicity, car color), highlighting its application to model introspection.

LoRA is shown to be more expressive and reliable than Textual Inversion for capturing intra-cluster diversity (Figure 4).

Empirical Results

DMA is empirically benchmarked against GANgealing, average-VAE, and state-of-the-art dataset distillation approaches (D$^4$M, MGD$^3$). DMA delivers minimal intra-prototype variance and strong semantic alignment without visual blurring, as opposed to GANgealing/Avg-VAE which, while consistent, yield low-quality prototypes, or dataset distillation baselines, which exhibit both semantic instability and artifacting.

Quantitative metrics include intra-prototype consistency (CLIP, LPIPS, DreamSim average distances), representativeness (mean prototype-to-sample distance), and perceptual quality (ImageReward). DMA achieves the best or near-best scores across all metrics, indicating that the method robustly reconciles representational sharpness, semantic centering, and output stability.

Figure 5: Qualitative comparison. DMA outputs are simultaneously sharper, more faithful, and more consistent than baselines across both concrete and abstract concepts.

Figure 6: Trade-off curves for consistency, representativeness, and perceptual quality. DMA achieves a Pareto-optimal balance where baselines must trade one for another.

DMA generalizes across both SD variants and architectures such as DiT by selecting corresponding semantic layers (Figures 7, 8). For SD variants, DMA reveals persistent biases (e.g., gender bias for "soldier", Venice-scene bias for "Italy") that are consistent across stylistic or demographic finetunings.

Figure 7: DMA prototypes from SD model variants visualize inherited biases and canonical representations within each model.

Figure 8: DMA applied to DiT transformer architectures, leveraging the final block as an analogue to U-Net h-space.

Analysis, Limitations, and Extensions

The reliance on external clustering for mode discovery transmits biases intrinsic to those encoders (e.g., CLIP or BLIP), but given clusters, the averaging procedure itself remains model-faithful. The DMA algorithm is computationally demanding, but its main hyperparameters ($K$, $N$, $t_\text{stop}$) allow explicit efficiency-consistency trade-offs. Ablation studies show that optimization across the full trajectory (rather than single-timestep, naive replacement, or pre-computed means) is essential for attaining high-fidelity mental averages, and that optimal CFG scales and sufficient $K$ are needed for consistency.

Implications and Future Directions

DMA provides previously unattainable access to the internal visual prototypes held by diffusion models, offering new tools for model interpretability, bias and fairness auditing, and model comparison (e.g., across fine-tunes and architectures). Practically, DMA could compress entire concept distributions into a single visual summary for rapid inspection or downstream tasks (e.g., dataset distillation, privacy-preserving summary, bias mitigation, or regularization).

Theoretically, the work advances understanding of semantic representation emergence and distribution across the temporal axis of diffusion models, and suggests research directions in defining/locating robust semantic spaces for model probing and alignment.

Conclusion

Diffusion Mental Averages represents a robust and general approach to deriving sharp, faithful, and consistent concept prototypes directly from off-the-shelf diffusion models. By leveraging progressive semantic trajectory alignment and external mode-conditioned guidance, DMA demonstrates superior performance to prior art in both qualitative and quantitative terms across a range of semantic, perceptual, and consistency metrics. The method not only exposes model biases and inner conceptual structure but also opens up new avenues in model interpretability and analysis. Future work may improve computational efficiency, extend mode discovery, or further integrate DMA into automated model auditing and fairness analysis workflows.