Semantic Measures for Idea Generation

• Semantic measures for idea generation are quantitative metrics that evaluate novelty, relevance, and feasibility using embedding-based and graph-theoretic models.
• They combine information-theoretic, frequency-based, and clustering methods to systematically assess creativity in both human and AI-driven ideation processes.
• These measures provide real-time guidance in diverse domains by distinguishing divergent and convergent thinking through dynamic semantic evaluation.

Semantic measures for idea generation comprise quantitative metrics, algorithmic protocols, and computational models designed to evaluate, monitor, and facilitate the creative process—particularly the production, refinement, and selection of novel, feasible, and contextually relevant ideas. These measures span from information-theoretic and knowledge-graph-based similarity indexes to embedding-driven novelty/diversity scores and frequency-based originality metrics, enabling systematic assessment and real-time guidance of human and AI-driven ideation processes across scientific, engineering, and design domains.

1. Theoretical Foundations for Semantic Evaluation in Idea Generation

The semantic evaluation of ideas traces its origins to theories distinguishing divergent thinking (novelty generation) from convergent thinking (usefulness, feasibility), and leverages both symbolic lexical resources (e.g., WordNet) and high-dimensional @@@@1@@@@. Early works operationalized semantic convergence and divergence via graph-theoretic constructs and information content measures over hierarchical ontologies. Georgiev & Georgiev anchored this research by demonstrating that temporal divergence of semantic similarity, increases in intrinsic information content (IC), and reductions in polysemy statistically predict the success of candidate ideas in real-world design conversations (Georgiev et al., 2021).

Formally, semantic similarity between concepts $x$ and $y$ is often computed via IC-weighted measures such as Lin similarity: $$\mathrm{sim}_{\mathrm{Lin}}(x, y) = \frac{2\,IC(\mathrm{LCS}(x,y))}{IC(x)+IC(y)}$$ where $IC(\cdot)$ is typically estimated intrinsically using vertex statistics from WordNet (e.g., Sánchez–Batet, Seco, Blanchard formulas) and $\mathrm{LCS}(x,y)$ is the lowest common subsumer (Georgiev et al., 2021). The trend lines of these measures, computed over a moving window during idea development, distinguish divergent (increasing IC, decreasing similarity) from convergent (decreasing IC, increasing similarity) ideation (Georgiev et al., 19 Jan 2025). These semantic signals are empirically aligned with human judgments of creativity and idea success.

2. Embedding-Based and Graph-Theoretic Measures

With the advent of large-scale pretrained models, ideas are increasingly projected into vector spaces where geometric operations express novelty, diversity, relevance, and clustering properties. Core measures include:

• Cosine Similarity: quantifies relevance between embeddings $e_a$, $e_b$:

$$\mathrm{sim}_{\cos}(e_a, e_b) = \frac{e_a^\top e_b}{\|e_a\|_2\,\|e_b\|_2}$$

• Euclidean Distance: signals innovation when $e_{\rm new}$ is distant from seeds:

$\mathrm{dist}_2(e_a, e_b) = \|e_a - e_b\|_2$

• Composite Scores: weighting novelty and relevance via acquisition functions (Bystroński et al., 18 Jul 2025).
• Cluster-Based Diversity: using clustering (e.g., DBSCAN, UMAP for dimensionality reduction) to find groupings of semantically similar ideas; outliers and cluster dispersion serve as proxies for creative divergence or uniqueness (Sankar et al., 2024).

The Graph of AI Ideas (GoAI) systematically encodes literature as a knowledge graph where citation edges are semantically and positionally weighted. Node embeddings, edge semantics (e.g., “based on,” “contrasts with”), node centrality, and path coherence steer AI models to generate more novel, yet anchored, research directions (Gao et al., 11 Mar 2025).

3. Human- and LLM-Judged Quality Indicators

Recent frameworks have introduced explicit, scalable scoring mechanisms aligned with classic creativity criteria:

• Novelty: Degree of difference from known/comparator ideas, evaluated via pairwise BERTScore, cosine/Euclidean distances, or LLM judge panels assigning 1–10 novelty scores (Guo et al., 2024, Ruan et al., 2024).
• Feasibility: Practical or scientific implementability, scored via LLM- or human-judged scales.
• Fluency: Quantity and diversity of ideas per prompt, assessed by measuring the semantic separation (low similarity) among generated outputs (Ruan et al., 2024).
• Flexibility: Consistency of generative performance across topics or domains, e.g., evaluated via the 30th percentile of combined novelty/feasibility scores over keywords (Ruan et al., 2024).
• Overlap: Degree of content match to a reference, via scalar LLM ratings or embedding scores (Guo et al., 2024).

IdeaBench, SCI-IDEA, and LiveIdeaBench instantiate these indicators in LLM-centric benchmarks, using systematic judge panels, prompt templates, and iterative “Aha-moment” detection via surprisal and embedding novelty thresholds (Guo et al., 2024, Keya et al., 25 Mar 2025, Ruan et al., 2024). Insight scores, such as

$$I(\mathrm{LLM},q) = \frac{1}{m}\sum_{i=1}^{m}\frac{r_{\mathrm{target}_i}|_q -1}{n}$$

quantify how frequently LLM-generated ideas surpass human references in novelty or feasibility (Guo et al., 2024).

4. Frequency, Clustering, and Distributional Measures

Purely frequency- or clustering-based measures, rooted in psychometric traditions, determine originality by idea rarity in a population. MuseRAG advances this at scale via embedding-based retrieval–augmented generation:

• Originality Scoring: Bucketing semantically indistinguishable ideas via LLM zero-shot bucketing, then calculating metrics such as rarity, uniqueness, and Shapley value:

$\mathrm{rarity}(B_{t,k}) = 1 - \frac{m_{t,k}}{N}$

• Adjusted Mutual Information and Intraclass Correlation: MuseRAG matches human inter-annotator agreement in clustering, validating that these frequency-informed measures are convergent with human notions of creativity and capture external correlates such as flexibility and self-identity (Bangash et al., 22 May 2025).

Distributional measures remain foundational for semantic relatedness computation, with metrics including cosine similarity, Jaccard, Dice, Lin mutual information, and Jensen-Shannon divergence (Mohammad et al., 2012). These measures are effective for both idea retrieval and as quantitative proxies for thematic proximity or creativity-potential in generative systems.

5. Semantic Navigation, Filtering, and Interactive Traversal

Innovative systems implement real-time semantic navigation and automated filtering to structure AI-assisted ideation:

• Semantic Navigation: Dual embedding spaces for problem/solution statements, with LLM-based parametric mappings (LoRA-fine-tuned). Idea generation alternates between nearest-neighbor selection and generative semantic mapping, achieving interactive depth-first traversal and controlled exploration (Sandholm et al., 2024).
• Filtering Metrics: Automated filters using relevancy ($R$), coherence ($C$), and explicit human alignment reward ($H$); these not only prune training sets for fine-tuning but also increase output quality and user engagement by quantifying semantic on-topicness, intra-idea smoothness, and human-preferred content (Sandholm et al., 2024).
• Iterative Prompting and Aha-Moment Detection: SCI-IDEA iteratively refines ideas, flagging candidates as “Aha” moments if their novelty and surprisal (embedding-based and LM-likelihood) cross empirical thresholds (Keya et al., 25 Mar 2025).

6. Empirical Validation, Limitations, and Future Directions

Empirical analyses across domains establish the predictive and human-aligned validity of these measures. In product design and research ideation benchmarks, statistically significant associations are observed between semantic-metric trends (e.g., rising IC, falling similarity) and idea success (Georgiev et al., 2021, Georgiev et al., 19 Jan 2025). Embedding-driven clustering and outlier analysis align with expert judgments and speed novice selection (Sankar et al., 2024).

Limitations include:

• Domain dependency of knowledge resources (e.g., WordNet coverage gaps).
• Sensitivity of frequency-based or bucketing methods to LLM prompt quality, embedding choice, and non-English contexts (Bangash et al., 22 May 2025).
• Automation challenges for non-verbal, visual, or multimodal ideation.
• Potential for overfitting or bias in LLM-based judge panels and prompt-driven scores, requiring dynamic normalization and leaderboard rotation (Ruan et al., 2024).
• The trade-off between generation of highly novel ideas and actual feasibility, with models tending toward one at the expense of the other (Guo et al., 2024).

Future research directions entail hybridization of distributional semantics with taxonomic and graph-based organization, multimodal extension, and integration with human–AI collaborative workflows (Georgiev et al., 19 Jan 2025, Gao et al., 11 Mar 2025). Real-time feedback systems, leveraging the statistical power and efficiency of IC and Lin/Sánchez–Batet similarity, are positioned for deployment in both design education and computer-augmented creative processes (Georgiev et al., 2021, Georgiev et al., 19 Jan 2025).

---

Summary Table: Principal Semantic Measures for Idea Generation

Measure / Metric	Mathematical Basis	Application/Frameworks
Information Content (Sánchez-Batet, Seco, Blanchard)	Taxonomic/graph-theoretic, intrinsic frequency	Divergence detection, specificity (Georgiev et al., 2021, Georgiev et al., 19 Jan 2025)
Lin similarity, Resnik, JC	IC/statistical/ontology	Trajectory tracking for divergence/convergence (Georgiev et al., 2021, Georgiev et al., 19 Jan 2025)
Cosine, Euclidean, KL divergence	Embedding-based, vector	Novelty/diversity scoring, clustering (Bystroński et al., 18 Jul 2025, Sankar et al., 2024)
BERTScore, Idea Overlap	Neural semantic similarity	Baseline similarity/overlap (Guo et al., 2024)
Frequency/Rarity-based originality	Population frequency	Automated human-aligned scoring (MuseRAG) (Bangash et al., 22 May 2025)
Judge (LLM/human) novelty, feasibility, fluency	Panel scoring, prompt-driven	IdeaBench, LiveIdeaBench, SCI-IDEA (Guo et al., 2024, Ruan et al., 2024, Keya et al., 25 Mar 2025)
Centrality, coherence (GoAI)	Weighted graph/semantic path	Graph-guided idea exploration (Gao et al., 11 Mar 2025)

These methodologies collectively underpin the contemporary computational paradigm for evaluating, guiding, and analyzing idea generation in both human-centric and AI-driven contexts.