SGHA: Evidence-Grounded Research Problem Discovery with Local Language Models

Abstract: Recent efforts toward fully automated AI scientists have demonstrated that language-model agents can generate hypotheses, execute experiments, and draft scientific manuscripts. However, during the early stages of research, when research problems are formulated, these AI scientists often rely heavily on proprietary frontier models. Their proposals are shaped by opaque parametric knowledge and by literature searches conditioned on the proposals themselves. Such knowledge is effectively a black box, and this dependence makes the evidential basis and validity of generated research problems difficult to audit and leaves the process vulnerable to model-specific hallucinations and biases. Furthermore, if proprietary research materials are transmitted to external APIs, the use of these models creates confidentiality, privacy, and data-governance concerns. We introduce the Structural Gap Hypothesis Agent (SGHA), a fully automated, corpus-first research-problem discovery system that runs entirely on a local LLM. SGHA structures a scientific literature corpus into evidence-linked paper objects and a typed evidence graph, detects unresolved structural patterns across papers, screens candidate gaps before formulation, and produces traceable research-problem families. In particular, it is able to output assumptions, objectives, success criteria, and remaining ambiguities. All LLM-based components of SGHA are executed using a locally served open-weight 9B LLM, without requiring proprietary frontier-model APIs. We compare SGHA with the AI Scientist-v2 idea formulation module in five machine-learning domains. Our results suggest that explicit corpus structure and evidence-constrained reasoning can support promising, inspectable research-problem formulation without relying on frontier models during generation or verification.

• The paper introduces SGHA, a training-free pipeline that extracts claims and limitations from 1,250 papers, detects cross-paper structural gaps, and converts verified patterns into traceable semi-formal research problems.
• SGHA produced 15 project families from 8,634 validated evidence tuples and achieved the highest overall formulation score, 5.99, outperforming local-Qwen and MOOSE-Star baselines while approaching Claude Opus.
• The results show that verification gates and structured evidence artifacts improve grounding and ambiguity control, but external novelty, scientific correctness, and eventual research impact remain untested.

Motivation and positioning

SGHA (Structural Gap Hypothesis Agent) is a training-free pipeline that formulates candidate research problems from unresolved cross-paper patterns in a scientific corpus, using only a locally served open-weight 9B LLM. The authors position the work against end-to-end "AI scientist" systems (AI Scientist v1/v2, ResearchAgent, AI co-scientist, Graph2Idea) whose ideation stages rely on proprietary frontier models and on literature searches conditioned on model-generated proposals. Two concerns motivate the design: the evidential basis of frontier-model ideation is difficult to audit, and transmitting sensitive research materials to external APIs raises confidentiality and governance issues.

The core conceptual move is to treat some research problems as structural gaps: patterns visible only when multiple papers are considered jointly — e.g., several methods failing under the same condition, or shared reliance on an unstated assumption. The motivating example is heavy-tailed regret in stochastic bandits: UCB achieved optimal $O(\log T)$ regret for bounded rewards, yet the heavy-tailed extension remained implicit in the literature until later robust-estimator work resolved it. SGHA aims to surface such gaps prospectively rather than generate ideas from open-ended prompting.

Pipeline architecture

The system proceeds through deterministic and LLM-mediated stages:

1. Evidence extraction. For each parsed paper, an LLM extracts paper-level objects (claims, limitations, failure conditions, assumptions) plus typed evidence tuples of the form $(\text{subject}, \text{relation}, \text{object})$ with provenance spans. Relations include assumes, fails_under, limited_by, contradicts, not_addressed_by, etc.
2. Deterministic validation. Tuples are filtered by relation vocabulary membership, span-to-text matching after normalization, removal of generic subjects ("our method"), and polarity normalization.
3. Typed evidence graph. Validated tuples populate a typed multigraph with node types for methods, assumptions, failure conditions, claims, datasets, metrics, etc.; edges retain passage-level provenance.
4. Motif detection. Deterministic queries over predefined motif families (assumption–failure, shared-failure, repeated-unaddressed-limitation, conflicting claims, sparse evaluation, theory–practice gap) instantiate candidate gaps.
5. Verification before formulation. A corpus-level screen discards trivial or already-solved candidates; surviving candidates undergo role-based review (support, skeptic, feasibility, mechanism agents, plus a critic). A deterministic hard gate requires all agent outputs, survival score $\geq 0.60$, non-rejecting critic with confidence $\geq 0.50$, and no parse failures.
6. Formulation and controlled ambition expansion. Verified gaps yield direct formulations close to the evidence; conservative/generalized/bold variants are then generated and filtered by an independent critic that rejects inflated or unsupported variants.
7. Consolidation and formalization. Critic-passing variants are grouped into project families and converted into semi-formal problem objects recording variables, assumptions (kept/relaxed/removed/questioned), objectives, success criteria, risks, falsification conditions, ambiguity flags, and source provenance.

Two auxiliary modes reuse this machinery: profile-conditioned generation (corpus construction steered by a researcher's literature context) and an optional evolutionary exploration branch that mutates graph-level seed representations for breadth; evolutionary outputs are reported separately from verification-gated families. Notably, when evidence is insufficient, SGHA reports a low-signal outcome rather than forcing outputs.

Experimental setup

Evaluation spans five ML domains — bandits, in-context learning, reasoning/test-time computation, offline RL, uncertainty estimation — each with 250 selected papers (OpenReview or arXiv sources). All SGHA components run on a frozen local Qwen3.5-9B with no task-specific fine-tuning. Baselines: AI-Scientist-v2 configured with the same local Qwen model (primary), AI-Scientist-v2 with Claude Opus (stronger-generator reference), and the public MOOSE-Star HC model. All methods are output-count-matched at 15 retained candidates, normalized into a common packet format with missing fields left as "not provided."

Judging uses five blinded LLM judges from different providers under a formulation-only rubric (ten criteria, 0–10 anchors, cap rules penalizing missing fields). The judges operate post hoc on frozen outputs and perform no external novelty checking — a constraint the authors acknowledge explicitly: judge scores assess formulation quality, not external novelty or correctness.

Results

Yield funnel. From 1,250 selected papers, 1,044 yielded valid extractions producing 8,634 evidence tuples; 836 motif hits produced 105 novelty survivors, 64 reviewed gaps, 39 verified gaps, 19 critic-passing variants, and finally 15 project families with 15 semi-formal statements. The funnel is deliberately lossy: breadth upstream, aggressive filtering downstream. Domain yields vary considerably (uncertainty estimation: 6 families; reasoning/test-time computation and offline RL: 1 family each).

Formulation quality. Under the blinded multi-judge rubric, SGHA achieves the highest overall score (5.99 vs. 5.84 for Claude Opus ideation, 4.55 for local-Qwen ideation, 2.00 for MOOSE-Star):

Method	Overall	Well-posed	Boundary	Formalizability	Grounding	Ambiguity
SGHA	5.99	5.77	6.45	5.51	7.43	7.61
AI-Scientist-v2 + Qwen	4.55	3.53	3.61	3.68	5.11	2.81
AI-Scientist-v2 + Claude Opus	5.84	4.88	4.75	5.21	2.97	3.57
MOOSE-Star	2.00	1.52	1.37	1.51	3.75	1.48

The pattern is consistent across domains. The strongest baseline (Claude Opus ideation) wins on clarity (7.15 vs. 6.77), technical specificity (6.95 vs. 5.89), nontriviality, and scope control — meaning a frontier generator still produces more polished prose — while SGHA dominates precisely on criteria tied to its artifact design: source grounding (7.43 vs. 2.97), ambiguity hygiene (7.61 vs. 3.57), assumption-boundary clarity, and formalizability. This is partly structural: SGHA emits formal problem objects and ambiguity flags by construction, whereas baselines leave those fields empty and are capped by the rubric's consistency rules. The authors are candid that this is a system-level comparison, not an information-matched ablation, since AI-Scientist-v2 retains Semantic Scholar retrieval while SGHA uses the bounded corpus.

Profile conditioning. Across three researcher profiles (LeCun, Hinton, Jordan), 439 papers yielded 9 verified gaps and 4 families. Personalized judging separates general formulation quality from profile fit: the Hinton candidate scores highest on alignment (8.00) and personalization overall (7.17), while the LeCun candidate has higher formulation quality but lower profile specificity. A representative Jordan-profile family connects variational inference and graphical-model themes to hierarchical identifiability limits in Gene Ontology protein annotation; notably, two of three ambition-expanded variants were rejected by the critic as inflated, and the accepted variant carries medium formalization confidence with requires_human_definition set to true.

Evolutionary exploration. The branch generates 2,908 evolved candidates, retaining 96 ranked hypotheses. Output-count-matched scoring yields an overall mean of 3.82, with strong source grounding (5.40) but weak well-posedness (2.80) and ambiguity hygiene (2.20). The contrast with main-path scores quantifies the value of the verification/formalization stages rather than treating them as formatting overhead.

Sensitivity analyses. In the Bandits domain, a 4B model produces verified gaps but zero final families (all variants fail the critic); a 27B model surfaces 19 verified gaps but consolidates to one family with the highest mean overall (6.67). Corpus-size variation from 50 to 250 papers shows the 50-paper run fails before family formation, while the best result occurs at 200 papers (5 families, mean 6.12) rather than at maximum corpus size. Both studies indicate output is governed by the filtering stages, not by scale alone — though these are single-domain diagnostics without variance estimates.

Limitations and open questions

The paper is explicit about scope. Evaluation is confined to formulation quality as assessed by LLM judges; external novelty, theoretical correctness, and eventual scientific impact are unmeasured, and the semi-formal statements are explicitly not finished theorem statements. Several caveats attach directly to the headline results:

• Structural advantage confound. SGHA's wins on grounding and ambiguity hygiene partially reflect that its artifacts contain fields baselines do not emit, and the rubric caps scores when such fields are absent. Whether human researchers would rate the underlying problems better, independent of artifact structure, remains open.
• Bounded-corpus verification. The corpus screen checks coverage only within the selected corpus; the paper concedes it is not an external novelty proof, so "verified" gaps may be solved elsewhere in the literature.
• Comparison asymmetry. The primary comparison matches output counts, not compute or information budgets; even the Claude Opus diagnostic acknowledges unmatched budgets.
• Small-sample judging. With 15 retained candidates per method and descriptive (non-statistical) reporting, differences such as SGHA's 5.99 vs. 5.84 overall rest on thin margins.
• Extraction bottleneck. Parsing failures are domain-dependent (137/250 extracted for reasoning/test-time computation vs. 248/250 for uncertainty estimation), and extraction errors propagate silently into motif detection; the paper does not quantify extraction precision against ground truth.

The central open question the paper leaves is whether verification-gated structural-gap discovery yields problems that experts judge externally novel and scientifically productive — the expert-evaluation protocol proposed in the predecessor PRISM work is still not carried out here.

Conclusion

SGHA demonstrates that a fully local, training-free pipeline can convert corpus-level structural patterns into traceable research-problem families, achieving the highest blinded-judge overall formulation score (5.99) among compared systems and large margins on grounding (+4.46 over the strongest baseline) and ambiguity hygiene (+4.04), while conceding that stronger generators produce clearer prose and that the evaluation does not establish external novelty. The contribution is less a new ideation capability than a demonstration that explicit corpus structure, hard verification gates, and structured artifacts can make automated problem formulation inspectable without proprietary models.