Distribution-Conditional Generation: From Class Distribution to Creative Generation (2505.03667v1)

Abstract: Text-to-image (T2I) diffusion models are effective at producing semantically aligned images, but their reliance on training data distributions limits their ability to synthesize truly novel, out-of-distribution concepts. Existing methods typically enhance creativity by combining pairs of known concepts, yielding compositions that, while out-of-distribution, remain linguistically describable and bounded within the existing semantic space. Inspired by the soft probabilistic outputs of classifiers on ambiguous inputs, we propose Distribution-Conditional Generation, a novel formulation that models creativity as image synthesis conditioned on class distributions, enabling semantically unconstrained creative generation. Building on this, we propose DisTok, an encoder-decoder framework that maps class distributions into a latent space and decodes them into tokens of creative concept. DisTok maintains a dynamic concept pool and iteratively sampling and fusing concept pairs, enabling the generation of tokens aligned with increasingly complex class distributions. To enforce distributional consistency, latent vectors sampled from a Gaussian prior are decoded into tokens and rendered into images, whose class distributions-predicted by a vision-LLM-supervise the alignment between input distributions and the visual semantics of generated tokens. The resulting tokens are added to the concept pool for subsequent composition. Extensive experiments demonstrate that DisTok, by unifying distribution-conditioned fusion and sampling-based synthesis, enables efficient and flexible token-level generation, achieving state-of-the-art performance with superior text-image alignment and human preference scores.

Analysis of Distribution-Conditional Generation for Creative Image Synthesis

The paper "Distribution-Conditional Generation: From Class Distribution to Creative Generation" by Fu Feng et al. introduces an innovative approach to enhance the creativity of text-to-image (T2I) diffusion models. Existing models in this domain are adept at generating realistic images aligned with textual descriptions, primarily based on the training data distributions. However, they struggle with generating novel and out-of-distribution concepts, largely because their generative creativity is bounded by their training data. While some methods attempt to enhance creativity by combining known concepts, these combinations often remain within established semantic boundaries.

The researchers propose a novel framework, Distribution-Conditional Generation, that redefines creativity in image synthesis as a function of class distributions. Unlike conventional methods that rely on fixed prompts or known reference concepts, this framework leverages probabilistic class distributions to condition image synthesis, thereby opening up new avenues for generating creative and semantically diversified images.

Methodology Overview

At the core of this research is the DisTok framework, an encoder-decoder model that translates class distributions into creative concepts. DisTok facilitates this transition through the following mechanisms:

• Distribution Encoding: The encoder ingests class distributions and projects them into a latent space.
• Creative Decoding: The decoder takes these latent representations and generates creative concept tokens, which drive the generation of novel images.
• Concept Pool and Sampling: DisTok maintains a dynamic concept pool that grows with newly generated tokens, allowing for continuous and iterative sampling and composition of more complex concepts over time.

DisTok employs a vision-LLM (VLM) to ensure that generated images align with the input class distributions. By periodically sampling latent vectors, DisTok refines its concept pool with novel tokens that adhere to the intended visual semantics, aligning the input distributions with the visually discernible semantics of the output images.

Experimental Validation and Results

DisTok's performance was validated using several benchmark tasks, including Distribution-Conditional Generation, Text Pair-to-Object (TP2O) tasks, and unconditional creative generation. Results indicate that DisTok not only outperforms state-of-the-art models like Stable Diffusion and Midjourney in generating images aligned with complex semantic distributions but also achieves a significant speedup over existing creative generation methods.

Key findings include:

• DisTok maintains semantic consistency across varying prompts, producing semantically coherent and visually integrated images reflecting detailed class distributions.
• The creative concepts generated by DisTok demonstrate high degrees of originality and aesthetic appeal, receiving favorable human evaluation scores.
• The framework excels in combining multiple concepts, yielding novel images that conventional models struggle to synthesize.

Implications and Future Prospects

The implications of this research stretch across both theoretical and practical domains. Theoretically, it introduces a robust framework for redefining creativity in artificial intelligence, emphasizing class distribution as a pivotal element of creative synthesis. Practically, it opens up avenues for applications where novel concept generation is valuable, such as in content creation, digital artistry, and virtual reality environments.

Future research could explore refining the latent space sampling strategies to enhance the diversity of generated concepts further and investigating the integration of additional semantic controls to refine creative outputs. Additionally, exploring the adaptability of DisTok's generated concepts in more diverse stylistic contexts could further demonstrate its versatility.

In summary, this research presents a significant step forward in the field of creative generative modeling, showcasing a system capable of producing novel and semantically sophisticated images beyond the constraints of existing training data distributions.