MAGenIdeas: Multi-Agent Research Ideation

• MAGenIdeas is a multi-agent LLM framework that iteratively generates, refines, and ranks research ideas using literature-based planning and multi-agent critiques.
• It applies combinatorial innovation theory by recombining methods, datasets, and metrics to produce structured abstracts from academic papers.
• Empirical results in NLP demonstrate that MAGenIdeas improves idea diversity, novelty, and quality compared to other AI-based ideation systems.

Searching arXiv for MAGenIdeas and closely related ideation / multi-agent idea generation frameworks. MAGenIdeas is a multi-agent, LLM-based framework for research idea generation that combines iterative planning, knowledge search, multi-agent critique, and tournament-style selection. It is grounded in combinatorial innovation theory, starts from a target ACL 2024 NLP paper and its references, constructs a virtual academic team from the paper’s authors, and iteratively generates, evaluates, and refines candidate ideas before producing structured abstracts. In experiments in the natural language processing domain, it improves diversity, novelty, and high-quality idea rate over AI-Researcher and NOVA, while the quality of its generated ideas falls between that of accepted and rejected ICLR 2025 NLP papers (Chen et al., 22 Apr 2026).

1. Concept and research setting

MAGenIdeas addresses research idea generation under two constraints identified in the source paper: the rapid growth of scientific literature, which increases the cost of knowledge filtering, and the tendency of existing LLM-based ideation systems to generate redundant, repetitive, and shallow directions. Its central design choice is to make LLMs behave less like a single brainstorming model and more like a team of collaborating scientists conducting a structured literature-based search (Chen et al., 22 Apr 2026).

The framework is defined as a multi-agent system built around an iterative planning and search process over papers, references, author profiles, candidate ideas, and multi-agent critiques. The implementation reported in the source paper is confined to NLP. Dataset construction starts from 675 ACL 2024 long papers and filters them to 144 papers with at least 10 citations, at least 20 references, and complete author information. These papers are enriched with ACL Anthology records, OpenAlex and Semantic Scholar metadata, and author profiles including research topics, institutions, publication lists, and citation statistics (Chen et al., 22 Apr 2026).

The operational unit of MAGenIdeas is a target ACL 2024 paper. For each target paper, the system generates 15 initial ideas, refines them over 3 iterations, and uses team sizes from 2 to 8 agents, with 5 target papers sampled per team size. In the reported experiments this yields 568 ideas in iteration 1, 656 in iteration 2, and 803 in iteration 3, for a total of 2,027 ideas (Chen et al., 22 Apr 2026).

2. Theoretical basis: combinatorial innovation

MAGenIdeas is explicitly grounded in combinatorial innovation theory. In this framing, innovation is treated as novel recombination of existing knowledge elements, and scientific ideas emerge from integrating concepts, methods, datasets, and related entities from different domains. The framework operationalizes this view by treating entities such as methods, tasks, datasets, metrics, and other scientific units as the basic materials of ideation (Chen et al., 22 Apr 2026).

The paper formalizes initial idea generation as

$R = f(P, L, T),$

where $P$ is the target paper, $L$ is its references, $T$ is a set of scientific discovery theories or methods, and $f$ is the LLM. In the implementation, $T$ is a catalog of 10 scientific discovery methods; the LLM selects 5 of them and generates 15 initial ideas with title, idea description, thought process, and rationale (Chen et al., 22 Apr 2026).

Iterative refinement is formalized as

$R_i = f(R_t, K, B),$

where $R_t$ is the current idea seed, $K$ is newly acquired knowledge from planned literature search, and $B$ is feedback from previous evaluation. This expresses the core recombination step: an agent revises a seed idea using both retrieved literature and multi-agent critique (Chen et al., 22 Apr 2026).

The entity-level analysis reported in the paper gives a more concrete view of how recombination unfolds. Using a fine-tuned scientific entity extractor with categories Method, Task, Metric, Dataset, and Other, the authors track overlap with the previous idea, overlap with retrieved references, and newly introduced entities. They report a three-stage pattern: turn 1 is “exploratory recombination,” with many new entities from the LLM (approximately 70%); turn 2 is “focused recombination,” with higher reuse from the previous idea (approximately 43%); and turn 3 is “path locking,” with high inheritance from the prior idea (approximately 78%) and very few new entities (Chen et al., 22 Apr 2026). This suggests that the framework expands the idea space early and then increasingly emphasizes deepening and selection.

3. Architecture and iterative workflow

MAGenIdeas is organized as a multi-agent pipeline with author-based agents, a search agent, an evaluation agent, and an abstract generation agent. The system begins by randomly selecting a target ACL 2024 paper, then generating initial ideas from the target paper, its references, and the scientific discovery methods. It subsequently constructs a virtual academic team from the real authors of the target paper (Chen et al., 22 Apr 2026).

The author-based agents $P$0 are initialized with real author backgrounds, including affiliations, topics, publication count, and citations. They function as idea generators and self-evaluators. The search agent plans literature search for each seed idea and retrieves papers through Semantic Scholar and related APIs. The evaluation agent compares candidate ideas pairwise and implements Swiss-system scoring. The abstract generation agent transforms final ideas into structured abstracts with objectives or problems, methods, expected results, and conclusions (Chen et al., 22 Apr 2026).

The reported workflow has four stages. First, dataset construction assembles target papers, references, and author profiles. Second, initial idea generation produces 15 ideas from the target paper $P$1, its references $P$2, and the discovery-method catalog $P$3. Third, iterative idea refinement repeats a loop of planning knowledge search, retrieving literature, generating revised ideas from $P$4, and ranking them via the Swiss-system tournament. Fourth, structured abstracts are generated for the surviving ideas (Chen et al., 22 Apr 2026).

The idea-generation prompt gives each author-agent a scientist identity tied to a real author profile and asks it to improve the seed idea or propose a new one by integrating the seed idea, “bad reviews,” and retrieved references. The required output includes a title, detailed idea, experiment plan, and scores for excitement, feasibility, and novelty on a 1–10 scale with rationales. The search agent outputs both a free-text “Thought” and a JSON search plan containing “Search Plan,” “Search Fields,” and “Search_Keywords.” The abstract generation agent outputs a title and an abstract of at most 300 words (Chen et al., 22 Apr 2026).

4. Planning-based search, multi-agent refinement, and ranking

A distinguishing feature of MAGenIdeas is that retrieval is planned rather than treated as a single-step keyword lookup. For each seed idea, the search agent analyzes the idea text and emits a structured search plan with relevant fields and keyword lists. The example reported in the source includes fields such as “Cognitive Biases in Human Language Processing” and “Machine-Generated Text Detection,” each paired with a list of keywords. These queries are executed through Semantic Scholar and similar APIs, and the resulting titles and abstracts are fed back to the author-agents as references (Chen et al., 22 Apr 2026).

The core iteration combines this planned retrieval with independent multi-agent refinement. At iteration $P$5, each author-agent receives a seed idea $P$6, negative reviews from previous comparisons $P$7, and literature $P$8 returned by the planned search. Each agent then produces one revised candidate idea $P$9. These candidates are pooled and passed to the evaluation agent (Chen et al., 22 Apr 2026).

Selection is performed through a Swiss-system tournament. Each idea participates in 5 pairwise matches, and each win contributes 1 point, yielding a total score $L$0. Ideas with score at least 5 are retained for the next iteration, together with the negative comments produced during comparison. The paper defines the “HighScoreRatio” as

$L$1

This makes tournament success the primary internal notion of quality (Chen et al., 22 Apr 2026).

The framework also defines semantic novelty and diversity metrics using all-MiniLM-L6-v2 sentence embeddings and cosine similarity. Novelty is computed against the top-10 retrieved papers for each generated idea: $L$2 with threshold $L$3. Diversity is computed against the other generated ideas: $L$4 These metrics are explicitly semantic rather than citation-based, because generated ideas do not yet have citation histories (Chen et al., 22 Apr 2026).

5. Empirical results and comparative evaluation

On ACL 2024 NLP papers, MAGenIdeas outperforms AI-Researcher and NOVA on all three reported automatic metrics. Its diversity is 0.898, compared with 0.867 for NOVA and 0.680 for AI-Researcher. Its novelty is 0.133, compared with 0.107 for NOVA and 0.067 for AI-Researcher. Its HighScoreRatio is 0.184, compared with 0.026 for NOVA and 0.013 for AI-Researcher (Chen et al., 22 Apr 2026).

The framework also generalizes across different LLM backbones. With DeepSeek-3.1, MAGenIdeas attains diversity 0.898, novelty 0.133, and HighScoreRatio 0.184. With GPT-4o, the reported values are 0.592, 0.151, and 0.140. With Qwen3-8B, they are 0.482, 0.183, and 0.154. The reported interpretation is that the framework works across heterogeneous backbones, while different models trade off diversity, novelty, and quality differently (Chen et al., 22 Apr 2026).

A further comparison places MAGenIdeas against real ICLR 2025 NLP submissions. In accepted-versus-rejected comparisons, accepted paper ideas have mean score 3.256 with standard deviation 1.095, while rejected paper ideas have mean score 1.744 with standard deviation 1.148, with $L$5 and $L$6. This is used to validate that the evaluation pipeline separates stronger from weaker paper ideas. In accepted-versus-generated comparisons, accepted ideas have mean 2.776 and generated ideas 2.224, with $L$7 and $L$8. In generated-versus-rejected comparisons, generated ideas have mean 2.689 and rejected ideas 2.311, with $L$9 and $T$0 (Chen et al., 22 Apr 2026). The resulting placement is that MAGenIdeas produces medium-quality ideas: better than rejected ICLR submissions on average, but weaker than accepted ones.

The paper also analyzes team size and iteration count. Diversity decreases as team size increases from 2 to 8, novelty remains relatively low and stable, and HighScoreRatio is highest and more stable for medium teams of size 4–7, then drops again at size 8. Across iterations, diversity peaks at round 2, novelty increases slightly, and HighScoreRatio continues to rise through round 3. The authors relate this to early expansion of the idea space followed by later trade-offs of diversity for quality (Chen et al., 22 Apr 2026).

A single-agent ablation keeps knowledge planning and search unchanged but sets the number of agents to 1. In this setting, diversity and novelty are higher in early iterations, but performance plateaus or declines later. The multi-agent version shows slightly lower diversity and novelty initially but higher and steadily increasing HighScoreRatio, supporting the claim that knowledge planning and search expand the space while multi-agent interaction improves quality and mitigates the plateau (Chen et al., 22 Apr 2026).

6. Limitations, failure modes, and position in the ideation literature

The source paper identifies several recurring failure modes. These include internal technical inconsistency, hallucinated claims or weak justification, under-specified integration of complex modular designs, and overly broad, unfocused proposals with no clear minimal core contribution. This means that high novelty or diversity does not guarantee feasibility or methodological coherence (Chen et al., 22 Apr 2026).

Additional limitations are domain specificity, since the reported evaluation is only in NLP; dependence on public metadata sources such as ACL Anthology, OpenAlex, Semantic Scholar, and OpenReview; and the fact that novelty, diversity, and LLM-judged quality do not fully capture technical consistency, feasibility, or long-term impact. The framework is also computationally expensive. In a 3-agent, 2-iteration configuration, the sequential full pipeline requires approximately 261.7 seconds total and 87.2 seconds per idea, with paper fetch, seed idea generation, literature search, and agent refinement as the main time components (Chen et al., 22 Apr 2026).

The ethical concerns recorded in the paper are primarily about plagiarism or “idea appropriation” and the possibility of flooding venues with AI-generated low-quality submissions. The framework is presented as an assistive system rather than a replacement for researchers, and the authors emphasize human oversight and critical evaluation (Chen et al., 22 Apr 2026).

Within the broader ideation literature represented in the supplied sources, MAGenIdeas belongs to a shift away from one-shot prompting and toward structured search or multi-agent orchestration. “Magellan” reframes creative generation as guided MCTS over a latent conceptual space with a semantic compass and an explicit value function over coherence, novelty, and progress (Chang, 24 Oct 2025). “MLE-Ideator” separates ideation from implementation and trains the ideation module with reinforcement learning using execution-based rewards (Zhang et al., 24 Jan 2026). “Supermind Ideator” encodes creative techniques as promptable “moves” organized into process-guiding workflows (Rick et al., 2023). These related systems differ in domain and mechanism, but collectively indicate that automated ideation is increasingly being treated as a problem of structured planning, feedback, and role specialization rather than unconstrained free-form generation.