LLMs can Realize Combinatorial Creativity: Generating Creative Ideas via LLMs for Scientific Research (2412.14141v2)

Abstract: Scientific idea generation has been extensively studied in creativity theory and computational creativity research, providing valuable frameworks for understanding and implementing creative processes. However, recent work using LLMs for research idea generation often overlooks these theoretical foundations. We present a framework that explicitly implements combinatorial creativity theory using LLMs, featuring a generalization-level retrieval system for cross-domain knowledge discovery and a structured combinatorial process for idea generation. The retrieval system maps concepts across different abstraction levels to enable meaningful connections between disparate domains, while the combinatorial process systematically analyzes and recombines components to generate novel solutions. Experiments on the OAG-Bench dataset demonstrate our framework's effectiveness, consistently outperforming baseline approaches in generating ideas that align with real research developments (improving similarity scores by 7\%-10\% across multiple metrics). Our results provide strong evidence that LLMs can effectively realize combinatorial creativity when guided by appropriate theoretical frameworks, contributing both to practical advancement of AI-assisted research and theoretical understanding of machine creativity.

• The paper proposes a novel framework using LLMs grounded in combinatorial creativity theory for generating scientific research ideas.
• A key technical contribution is a multi-level generalization retrieval system combined with a structured combinatorial process to connect disparate concepts.
• Empirical evaluation shows the framework outperforms a baseline in generating ideas semantically similar to actual research papers, validated by qualitative analysis.

This paper, "LLMs can realize combinatorial creativity: generating creative ideas via LLMs for scientific research," proposes a novel framework for using LLMs to generate creative scientific research ideas, grounded in the theory of combinatorial creativity. It addresses a key limitation of existing LLM-based idea generation systems: their lack of grounding in established creativity theories. The authors focus specifically on Boden's concept of combinatorial creativity, which involves combining existing concepts in novel and unexpected ways.

The core contributions of the paper are:

1. A Theoretical Framework: The paper explicitly maps principles of combinatorial creativity onto LLM-based idea generation. It draws heavily on the "Four P's" framework of creativity research (Person, Process, Product, Press) and Boden's theory of conceptual spaces, focusing on combinatorial creativity within the "Process" perspective.
2. A Generalization-Level Retrieval System: A novel retrieval system is introduced to facilitate cross-domain knowledge discovery. Unlike traditional retrieval methods (like RAG) that often rely on surface-level similarity, this system uses a semi-structured "ideation format" to store and retrieve concepts at multiple levels of abstraction (L1-L4, from domain-specific to universal principles). This enables the LLM to connect ideas from seemingly disparate domains by matching them at higher levels of abstraction. The retrieval system employs a two-stage pipeline: analysis of the problem statement (using structured prompts to an AI agent), and similarity search via OpenAI's text-embedding-3-large, calculating cosine similarities at each generalization level.
3. A Structured Combinatorial Process: The paper outlines a two-stage process for combining retrieved concepts to generate novel ideas.
   • Stage 1 (Parallel Processing): AI agents analyze retrieved innovations at each generalization level from three perspectives: component analysis (breaking down innovations), cross-domain application (identifying potential applications in new contexts), and building block assessment (evaluating components as foundations for new solutions).
   • Stage 2 (Integration): An integration agent synthesizes the analyses from Stage 1, focusing on feasibility and innovativeness. It generates solutions characterized by a problem structure, design rationale, universal principles, and key mechanisms.
4. Empirical Evaluation: The authors evaluate their framework using the OAG-Bench dataset, which contains scientific papers and their key references. They compare their system's generated ideas to the actual innovations presented in the target papers, using the references as the knowledge base. They use a baseline that generates ideas directly from a problem statement, without the multi-level retrieval and combinatorial process. Evaluation metrics include semantic similarity scores (using allenai-specter embeddings) between generated ideas and target papers, focusing on: Problem Structure Similarity (PS-Sim), Design Rationale Similarity (DR-Sim), Universal Principle Similarity (UP-Sim), and Key Mechanism Similarity (KM-Sim). The results demonstrate that their framework consistently outperforms the baseline across all metrics, with particularly strong improvements in stability and high-end performance.
5. Qualitative Analysis: The paper includes a qualitative analysis of three representative cases, comparing generated solutions to actual research developments. This analysis reveals strong alignment between the generated ideas and the core concepts, technical designs, and even implementation details of the target papers.

The paper's related work section provides a comprehensive overview of computational creativity research, covering the "Four P's" framework in detail and discussing existing work on LLM-based idea generation. It highlights the gap between current LLM applications and established creativity theory, motivating their theoretically-grounded approach.

In conclusion, the paper argues that LLMs can effectively realize combinatorial creativity when guided by theoretical frameworks. The framework presented contributes to both the practical advancement of AI-assisted research and the theoretical understanding of machine creativity. The paper concludes by discussing future research directions, including extending the framework to other types of creativity (exploratory and transformational), developing more comprehensive evaluation metrics, and incorporating real-time feedback mechanisms. The appendix provides complete comparison samples between generated solutions and actual research outcomes for all 87 papers.