- The paper shows that proprietary data raises curated competitor recall from 0.38 with skills and public tools to 0.96, including a 0.93 long-tail recall rate.
- The paper finds that reasoning skills improve calibration on validated white-space opportunities, raising tier-in-range accuracy from 0.80 to 0.89, but cannot overcome missing evidence.
- The paper demonstrates that data-enabled agents achieve 7.43 informed decision-quality points versus 1.76 for plain agents and 2.57 for skills-enabled agents, while all conditions avoid false-GO decisions.
Motivation and central question
Evaluations of AI Scientist agents typically attribute performance to model quality, prompting, or reasoning scaffolds. This paper argues that in specialized, knowledge-intensive domains the binding constraint may instead be the evidence substrate — the private, curated, long-tail record over which the agent is permitted to reason. The authors test this hypothesis in drug-asset valuation, where a go/watch/no-go call on a (target, modality, indication) triple requires synthesis of target biology, competitive pipeline, clinical precedent, and business-development (BD) intelligence, and where per-approval costs of roughly $2.6B make early licensing decisions high-stakes (2606.09556).
The paper's contribution is a controlled three-arm ablation that separates two enhancements practitioners routinely conflate: (1) skills and public structured tools — scoring playbooks, deterministic guardrails, citation verification, an objectivity charter, adversarial red-team review, and public APIs (ClinicalTrials.gov, PubMed, OpenTargets, OpenFDA); and (2) proprietary data — the Noah AI corpus of 5,322 programs, 16,547 trials, and 476 deals aggregated from 50,000+ pharmaceutical-company websites, journals, conferences, and news media.
Experimental design
All three arms run the same backbone model (Claude Code with Opus 4.8 MAX) under equal time and scope budgets, so model capability is held constant while tool and data access vary:
- A (plain): web search only; no rubric, skills, or tools — described as a fair baseline rather than a crippled one.
- B (+skills +public tools): the full 14-dimension playbook, public APIs, verifier, objectivity charter, and red-team pass, with proprietary data starved by pointing
LOCAL_DATA_DIR at an empty directory — an environment-level control rather than a prompt-level suggestion.
- C (+data): identical to B but with Noah AI exports present and queried first at target level (competition being a target×modality property).
The benchmark comprises 13 single-target assets stratified into five decision archetypes chosen to expose specific failure layers: crowded me-toos (S1), validated-mechanism white-space opportunities (S2), unvalidated-target false-positive traps (S3), deal-sensitive assets (S4), and biological voids such as antibodies against intracellular targets (S5). Bispecific assets are excluded because their competitor-truth denominator is not comparable. The scored rollup contains 28–30 asset–seed cells per condition after excluding incomplete cells.
A candid methodological point deserves emphasis: the gold answer keys were drafted from the same proprietary commercial-intelligence family available to C (Noah AI/Citeline/BMT exports) and then expert-audited. The design is therefore intentionally circular with respect to C's data access — it directly tests whether an agent can recover a curated diligence record, but it is not a database-independent estimate of real-world competitors or clinical payoff. The authors state this plainly and retain an R_union hook for independently sourced extensions, though web_extra is empty in this study.
Metrics
The metric suite is organized along three axes. Factual grounding includes recall against the curated gold competitor set (R_gold), long-tail recall restricted to non-web-famous programs (preclinical, regional/China, unregistered), deal recall against curated marquee transactions (e.g., Merck/Prometheus $10.8B, J&J/Momenta $6.5B), and over-generation counts explicitly distinguished from hallucination rates because the gold set is non-exhaustive. Methodological discipline covers objectivity (eight codes scored 0–4 by a hard-blind three-judge panel with provenance scrubbed and reports shuffled) and key-fact coverage. Decision quality includes tier-in-range accuracy, false-GO rate, blind-panel decision quality (DQ), and two novel constructs: informed decision-quality (DQ × gold-coverage) and coverage-limited decision utility, reported under product, hard-ceiling, pessimistic-min, and geometric proxies.
Two accounting choices matter for interpretation. First, factual recall uses capability-superset accounting: each arm is credited with the cumulative union of its own and all lower arms' findings, making recall monotone in capability and deliberately generous to B and C. Second, blinding is complete only between B and C; A's free-form report format is structurally distinguishable from B/C scorecards, so A's judge scores are read alongside auto-metrics rather than as a perfectly blinded comparison.
Headline results
The overall pattern is a clean dissociation between calibration gains and factual ceiling:
| Metric |
A plain |
B +skills/tools |
C +data |
| R_gold |
0.25 |
0.38 |
0.96 |
| Long-tail recall |
0.26 |
0.30 |
0.93 |
| Deal recall (capacity) |
0.81 |
0.88 |
0.92 |
| Objectivity (/4) |
3.16 |
3.30 |
3.60 |
| Tier-in-range |
0.80 |
0.89 |
0.89 |
| Blind-panel DQ (/10) |
7.01 |
6.96 |
7.65 |
| Informed DQ (DQ × coverage) |
1.76 |
2.57 |
7.43 |
| False-GO rate |
0.0 |
0.0 |
0.0 |
Three findings carry the paper's argument. First, 60–75% of the curated competitive record is unreachable without the proprietary data: even under generous capability-superset accounting, B recovers only 0.38 of the gold record versus C's 0.96, and the long-tail gap is starker still (0.30 vs. 0.93). Long-tail recall is presented as the cleanest single signal of data value, since preclinical, regional, and unregistered programs are precisely what ad-hoc web search fails to surface. Second, raw verdict-soundness is insensitive to completeness: A and B score nearly identically on blind-panel DQ (7.01 vs. 6.96) despite a roughly fourfold recall gap, because a judge reading report prose cannot see missing competitors. This motivates informed-DQ, where C reaches 7.43 versus 1.76/2.57 — approximately a 4× advantage. Third, the ceiling analysis makes the bottleneck explicit: with B's 0.383 coverage on utility-evaluable cells, even a hypothetically perfect DQ = 10 report would be capped at 3.83 informed-DQ, below C's observed 7.43. The ordering survives min and geometric utility proxies, and a paired sensitivity analysis on 27 fully paired cells confirms it under raw autonomous retrieval as well (C raw R_gold 0.929 vs. A 0.262 / B 0.233).
Notably, no arm commits the cardinal error of a false GO — including on S5 biological-void cases, where even the plain agent recognizes that an antibody against an intracellular target is implausible. The discriminating failure mode on this benchmark is under-valuation, not recklessness.
The calibration finding
The most decision-relevant effect of the non-proprietary layer appears in stratum S2. Across all three seeds, the plain agent returns no_go on every validated-mechanism white-space asset (IFNAR1 × scleroderma and IFNAR1 × lupus-nephritis; tier-in-range = 0.0), exhibiting indiscriminate pessimism: it cannot distinguish "avoid because crowded" from "pursue because the mechanism is validated elsewhere and the indication is open." Arm B partially corrects this (tier 0.5), via the scientific-rationale skill crediting mechanism validation and calibrated thresholds elevating the asset, with panel DQ improving from 4.9 to 5.8 (C reaches 6.3). This is the clearest evidence that the skills layer changes the decision itself rather than merely its presentation — while simultaneously demonstrating that calibration alone cannot compensate for a missing evidence base.
Boundary conditions of the claim
Deal recall marks where the data-dependence claim weakens. Marquee transactions are unusually web-visible, so arm A already recovers 0.81 and the proprietary feed adds only 4 points on top of B's public-API increment, under capacity accounting. On famous assets and textbook biology, the open-web baseline can be strong. The data dependence concentrates exactly where expected: long-tail pipeline programs, regional or preclinical assets, sparse indications, and diligence questions requiring "who else exists" rather than "what is the headline precedent."
Threats to validity
The authors are explicit about several limitations. Beyond the gold-set circularity noted above, there is no clean payoff ground truth — asset value settles over years, so headline claims are anchored on the least-circular axes (coverage and long-tail recall). LLM-judge bias is mitigated by fixed rubrics, provenance scrubbing, independent-then-compare scoring, and supplied factual ground truth, but small inversions (S5 DQ C < A; S3 DQ B < A) are within noise and reported honestly. Capability-superset recall is a capacity estimate, not single-run autonomous retrieval behavior, though the central claim is robust to this generosity. Informed-DQ is acknowledged as a proxy rather than a canonical utility theorem, defended by the robustness variants. Finally, strata are small and unevenly covered — S4 has partial B/C scored-cell coverage (3/1/2 cells), and the load-bearing S2 calibration claim rests on two assets at three seeds, though the 0.0-vs-0.5 tier gap exceeds seed noise. Cost telemetry was collected but not made load-bearing due to incomplete per-condition accounting.
Conclusion
This study supports a specific thesis: for knowledge-intensive AI Scientist agents, capability is bounded by the evidence substrate. Reasoning scaffolds and public tools function as control layers — correcting systematic pessimism on white-space opportunities and lifting objectivity modestly — but they cannot manufacture the missing long-tail record, and their best possible decision quality is capped well below what the data-enabled system achieves. The practical prescription is "skills on top of the right data," not data instead of skills. Open questions include rigorous cost accounting across arms, extension to bispecific assets with comparable denominators, and re-scoring against independently sourced gold keys once web_extra annotations exist.