Large Language Models as Innovators: A Framework to Leverage Latent Space Exploration for Novelty Discovery (2507.13874v1)

Abstract: Innovative idea generation remains a core challenge in AI, as LLMs often struggle to produce outputs that are both novel and relevant. Despite their fluency, LLMs tend to replicate patterns seen during training, limiting their ability to diverge creatively without extensive prompt engineering. Prior work has addressed this through domain-specific heuristics and structured prompting pipelines, but such solutions are brittle and difficult to generalize. In this paper, we propose a model-agnostic latent-space ideation framework that enables controlled, scalable creativity by navigating the continuous embedding space of ideas. Unlike prior methods, our framework requires no handcrafted rules and adapts easily to different domains, input formats, and creative tasks. This paper introduces an early-stage prototype of our method, outlining the conceptual framework and preliminary results highlighting its potential as a general-purpose co-ideator for human-AI collaboration.

• The paper introduces a latent space exploration framework that enhances LLM creativity by generating novel ideas from seed embeddings.
• The methodology employs interpolation, cross-modal projection, and iterative feedback using SRF-Embeddings-Mistral and Mistral 7B, improving originality and fluency.
• Empirical results show modest yet significant gains on benchmarks like AUT, highlighting both the promise and limitations of the current interpolation approach.

Latent Space Exploration for LLM-Driven Novelty Discovery

This paper presents a model-agnostic framework for enhancing creative idea generation in LLMs by leveraging systematic exploration of the semantic latent space. The approach is motivated by the limitations of current LLMs, which, despite their fluency, tend to produce outputs that are derivative of their training data and lack genuine novelty. The proposed framework circumvents the need for domain-specific heuristics or prompt engineering by operating directly in the continuous embedding space of ideas, enabling scalable and adaptable creativity augmentation across diverse domains.

Framework Architecture and Methodology

The core pipeline consists of the following modular components:

1. Semantic Encoder: Transforms seed ideas or prompts into fixed-dimensional latent embeddings using a frozen, domain-agnostic encoder.
2. Latent Explorer: Generates new candidate embeddings via interpolation, extrapolation, or noise-based perturbations in the latent space. The current implementation focuses on interpolation between seed embeddings.
3. Cross-Modal Projector: Maps latent vectors into the token embedding space of a decoder LLM using a learned projection (xRAG-style), enabling the LLM to condition on these vectors as virtual tokens.
4. Decoder LLM: Generates natural language descriptions from the projected embeddings, effectively decoding latent points into textual ideas.
5. Evaluator LLM: Scores generated ideas against creativity rubrics, primarily focusing on originality and relevance.

A feedback loop allows high-scoring ideas to be reincorporated as new seeds, supporting iterative refinement and expansion of the idea manifold.

Implementation Details

• Encoder: SRF-Embeddings-Mistral is used for semantic encoding.
• Decoder: Mistral 7B serves as the generative LLM.
• Projector: An MLP-based projector, as in xRAG, bridges the encoder and decoder spaces.
• Evaluation: GPT-4o is employed as an LLM-based judge, but only for scoring, not generation.
• Exploration Strategy: Interpolation is performed with $\lambda \sim [0.45, 0.55]$ between random seed pairs; extrapolation and noise-based perturbations are proposed for future work.
• Filtering: Only ideas with high originality (score ≥ 4) and relevance are retained, resulting in aggressive rejection of low-quality outputs.

Empirical Results

The framework is evaluated on standard creativity benchmarks, including the Alternative Uses Test (AUT), Instances, Similarities, and Scientific ideation tasks, using the LLM Discussion method as a baseline. The following table summarizes the key results:

Benchmark	Method	Originality (Mean)	Elaboration (Mean)	Fluency (Mean)	Flexibility (Mean)
AUT	Ours (2 iter)	4.160	3.152	12.150	11.467
AUT	LLM Discussion	4.148	3.116	11.108	11.525
Instances	Ours	4.150	2.108	11.908	10.308
Instances	LLM Discussion	4.149	2.117	11.233	10.575
Similarities	Ours	3.467	1.744	8.960	13.725
Similarities	LLM Discussion	3.464	1.744	8.733	13.625
Scientific	Ours	3.518	2.059	7.508	8.333
Scientific	LLM Discussion	3.510	2.049	7.217	8.358

The framework consistently yields improvements in originality and fluency over the LLM Discussion baseline, with the most pronounced gains observed in the AUT and Instances tasks. The improvements, while statistically significant, are modest, reflecting the conservative filtering strategy and the limitations of interpolation-based exploration. Flexibility scores are slightly reduced, likely due to the semantic blending inherent in interpolation, which may reinforce broader categories rather than introducing new ones.

Theoretical and Practical Implications

The latent-space ideation framework demonstrates that systematic navigation of the embedding manifold can unlock creative potential in LLMs that is otherwise inaccessible through prompt engineering or multi-agent discussion alone. By decoupling the creative process from domain-specific rules, the approach is highly adaptable and compositional, supporting a wide range of input formats and ideation tasks. The iterative feedback mechanism enables the system to self-improve, continually expanding the set of high-quality, novel ideas.

From a theoretical perspective, the work aligns with recent advances in manifold mixup and latent space augmentation, extending these concepts to the domain of computational creativity. The results suggest that the semantic structure of LLM embedding spaces is sufficiently rich to support meaningful interpolation and recombination of ideas, provided that appropriate projection and decoding mechanisms are in place.

Limitations and Future Directions

The primary limitation of the current prototype is its reliance on interpolation, which, while effective for generating coherent blends, may not fully exploit the creative potential of the latent space. The aggressive rejection policy, while ensuring high output quality, results in low yield and may discard valuable outliers. The evaluation protocol is also constrained by the use of LLM-based judges, which may introduce biases or fail to capture nuanced aspects of creativity.

Future research directions include:

• Advanced Exploration Strategies: Incorporating swarm-based or evolutionary algorithms for more diverse and efficient latent space traversal.
• Human-in-the-Loop Feedback: Integrating real-time human evaluation to guide exploration and selection.
• Domain-Specific Metrics: Developing lightweight, objective evaluators tailored to specific creative domains.
• Generalization Beyond Text: Extending the framework to multimodal ideation tasks, such as product design or scientific hypothesis generation.

Conclusion

This work establishes a principled, model-agnostic approach for augmenting LLM creativity via latent space exploration. The framework's adaptability, compositionality, and iterative refinement capabilities position it as a promising foundation for future AI co-ideators. The empirical results validate the potential of latent space operations to enhance originality and fluency in idea generation, while highlighting the need for more sophisticated exploration and evaluation techniques to fully realize the creative capacity of LLMs.