MotivGraph-SoIQ: Grounded Socratic Ideation

• The framework introduces a dual-agent approach combining literature-grounded knowledge graphs with adversarial questioning to rigorously refine LLM-generated ideas.
• Empirical validation reveals significant improvements in idea novelty, experimental rigor, and motivational alignment compared to existing LLM ideation methods.
• It employs embedding-based clustering and semantic similarity measures to ensure traceability of ideas, mitigating confirmation bias in academic ideation.

MotivGraph-SoIQ is a framework integrating motivational knowledge graphs with Socratic dual-agent dialogue for enhanced ideation using LLMs. Designed to address persistent limitations in LLM-driven academic ideation—specifically, the lack of explicit grounding and the tendency toward confirmation bias—the system structurally grounds creative reasoning in literature-derived motivation while iteratively refining ideas under adversarial questioning. Empirical validation demonstrates improvements in ideation quality, novelty, rigor, and motivational alignment compared to state-of-the-art LLM-based ideation approaches (Lei et al., 26 Sep 2025).

1. Motivation and Problem Definition

LLMs accelerate academic ideation but remain fundamentally limited by two critical challenges. First, grounding of generated ideas is weak, as LLMs operate on probabilistic knowledge often resulting in hallucinated or superficial motivational justifications. Second, confirmation bias arises during self-reflection, with LLMs tending to reinforce their own initial proposals rather than surface flaws or challenge questionable assumptions. MotivGraph-SoIQ directly addresses these barriers by combining a motivational knowledge graph ("MotivGraph"), which enforces traceability of ideas to documented research, with a Q-Driven Socratic Ideator, a dual-agent system simulating adversarial mentoring (Lei et al., 26 Sep 2025).

2. Motivational Knowledge Graph (MotivGraph)

MotivGraph is structurally formalized as

$G = (V, E)$

where $V = P \cup C \cup S$ contains nodes for "problems" ($P$), "challenges" ($C$), and "solutions" ($S$), derived from literature via the SciMotivMiner extractor: $\{(P_i, C_i, S_i)\}_{i=1}^n = SMM(\text{Paper})$ Edges ($E$) encompass:

1. parent_of relations within node types (enabling hierarchical structure);
2. directed problem–challenge ($p \to c$);
3. directed challenge–solution ($c \to s$) links.

Node consolidation employs embedding-based clustering and LLM-verified merging, producing hierarchical abstractions. Although base edges are unweighted, semantic similarity

$w(u, v) = \cos(\text{embed}(u), \text{embed}(v))$

can guide retrieval-based reasoning.

During ideation, the system ensures traceability by requiring generated content to hyperlink referenced nodes, anchoring idea components in explicit problem–challenge–solution (P–C–S) sequences and thereby supplying intrinsic motivation cues. This grounding strategy mitigates hallucination and knits creativity tightly to explicitly documented research pathways (Lei et al., 26 Sep 2025).

3. Q-Driven Socratic Ideator

The Q-Driven Socratic Ideator formalizes a dual-agent dialogue between a Researcher LLM and a Mentor LLM:

• Researcher: accesses MotivGraph, uses external search (Semantic Scholar), and formulates candidate ideas.
• Mentor: acts as a domain-skeptical supervisor, interrogating the Researcher along axes of innovation, feasibility, and rationality but never supplying answers.

The ideation process proceeds in two phases:

1. Exploration: The Researcher gathers context from MotivGraph and literature, performs random node selection to inject novelty, and produces initial idea $Idea_0$.
2. Deliberation: Over $N$ rounds, the Mentor issues questions ($Q_k$), the Researcher responds and revises ($Idea_k$), with the process continuing until the Mentor elects to stop and accepts or rejects the proposal.

Formally,

$$Idea_k = f_{\text{Res}}(Idea_{k-1}, I_k, \text{Exploration}),\qquad I_k = (\text{Question}_k, \text{Answer}_k)$$

and the process terminates when

$$Eval = e_{\text{Mentor}}(Idea_{\text{final}}, \text{DialogueHistory}) \in \{\mathrm{ACCEPT},\,\mathrm{REJECT}\}.$$

By forcing explicit justification in response to critical inquiry, this approach systematically reduces confirmation bias; unreconciled flaws prompt revision or abandonment of ideas (Lei et al., 26 Sep 2025).

4. Integration Workflow

The end-to-end workflow orchestrates MotivGraph-driven exploration and Socratic refinement, illustrated as follows:

for each topic T in ICLR25_topics: # 1. Exploration nodes = Researcher.node_search(T) relations = Researcher.node_relation(nodes) random_triples = Researcher.get_random_nodes() Idea_0 = Researcher.generate_idea(nodes, relations, random_triples) # 2. Socratic Deliberation for k in 1..N_max: Q_k = Mentor.generate_question(Idea_{k-1}, DialogueHistory) A_k = Researcher.respond(Q_k, MotivGraph, semantic_search) Idea_k = Researcher.update_idea(Idea_{k-1}, A_k) if Mentor.decides_to_stop(Idea_k): break # 3. Final evaluation if Mentor.eval(Idea_k) == ACCEPT: record Idea_k else: discard

Key API operations include targeted and random MotivGraph queries, support for semantic search, and interactive dialogue. The workflow’s structure enforces rigorous idea improvement cycles, integrating literature grounding and adversarial refinement (Lei et al., 26 Sep 2025).

5. Empirical Evaluation and Results

Experimental validation uses 100 topics sampled from clustered ICLR 2025 paper titles, with ground-truth ideas extracted and standardized. Each method, including strong baselines, generates multiple ideas per topic. Evaluation is conducted using:

• Diversity:

$$\mathrm{Diversity} = 1 - \frac{1}{M^2} \sum_{i,j} \mathrm{sim}(Idea_i, Idea_j)$$

• LLM-evaluator scores (Fast-Reviewer): Novelty, Experiment, Motivation (0–10 scale).
• Swiss-tournament ELO ranking via pairwise LLM judgments.
• Human evaluation: manual scoring and ELO, using DeepSeek-V3 outputs.

Quantitative comparisons are summarized below.

Method	Diversity	Novelty	Experiment	Motivation
MotivGraph-SoIQ (V3)	0.45	8.39	8.64	8.70
Next best baseline	0.42	7.61	7.23	7.61

MotivGraph-SoIQ achieves statistically significant improvements ($p < 0.01$) in all reported metrics:

• ELO (LLM-evaluator): Novelty +28 pts, Experiment +33 pts, Motivation +45 pts, mean +38 pts overall.
• Human evaluator gains (DeepSeek-V3): Novelty +7.98%, Experiment +5.56%, Motivation +5.56%.

These results confirm superior performance in diversity, motivational coherence, and experimental rigor relative to leading alternatives (Lei et al., 26 Sep 2025).

6. Comparative Performance

Against six contemporary ideation frameworks (including AI-Scientist-v2, AI-Researcher, SciPIP, CycleResearcher, ResearchAgent), MotivGraph-SoIQ demonstrates:

• Highest automatic LLM-evaluator score improvements: Novelty (+0.78), Experiment (+0.25), Motivation (+0.49) on a 10-point scale.
• Superior diversity (0.45 vs. ≤0.42).
• Outperformance in ELO ranking by +38 pts on average.
• Higher manual and LLM-based scores in human evaluations.

This consistent dominance across multiple axes highlights MotivGraph-SoIQ's efficacy in overcoming limitations intrinsic to traditional LLM-driven ideation (Lei et al., 26 Sep 2025).

7. Limitations and Prospective Directions

Current implementation displays several restrictions:

• Domain coverage: MotivGraph is AI/ML-centric; breadth across other sciences is limited.
• Resource constraints: Empirical results are based on 100 topics and three LLMs; broader scaling is desirable.
• Mentor agent knowledge: Lack of subdomain expertise occasionally yields weak questioning.

Proposed future work includes:

• Broadening MotivGraph coverage to domains such as medicine and physics; initial tests suggest retained performance gains.
• Scaling the number and diversity of topics and LLMs (e.g., GPT-4, LLaMA variants) to test generalizability.
• Systematic exploration of alternative dialogue formats (e.g., multi-mentor debates, counterfactual interrogation) to enhance bias mitigation and ideational diversity.
• Integration of experimental feedback loops to move beyond plan-based validation.

These directions aim to enhance methodological robustness, cross-disciplinarity, and real-world relevance (Lei et al., 26 Sep 2025).