Proposal-Stage Scientific Triage

• Proposal-stage scientific triage is a structured, algorithmic screening method for research proposals aimed at maximizing innovation and resource efficiency.
• It integrates techniques such as merit scoring, distributed peer review, and AI-driven evaluations to balance workload and enhance accuracy.
• Future directions include threshold-based models, LLM assistance with human oversight, and simulation-driven risk calibration to further refine proposal selection.

Proposal-stage scientific triage refers to the structured, algorithmic prioritization and screening of scientific research proposals before allocation of review resources or funding. This process, distinct from post hoc evaluation of completed work, aims to maximize expected innovation, optimize reviewer and applicant workload, and increase transparency and reproducibility in scientific funding and facility allocation. Triage methods range from classical merit review and threshold optimization, to distributed peer review, agent-based AI systems, and taxonomy-guided early rejection or refinement mechanisms. The field draws from formal decision theory, empirical analysis, simulation studies, and the latest work on LLM-assisted proposal assessment.

1. Theoretical Foundations and Optimization Models

Early formalizations of proposal-stage triage cast the selection process as a constrained optimization problem over proposal merit scores. Navarra’s model (Navarra, 2015) regards each proposal $i$ as characterized by a normalized ex-ante innovation score $x_i \in [0,1]$ and employs a monotonic utility function $v(x)$ representing innovation density. The triage decision is framed by the cost function

$J(t) = \int_t^1 p(x) v(x) dx + \int_0^t p(x) dx$

where $p(x)$ is the empirical or parametric density of proposal scores and $t$ is the triage threshold. The unique maximizer $t^*$ solves $v(t^*) = 1$ (for properly normalized models).

This approach is extensible to budget-constrained settings via a 0/1 knapsack formulation, or solved approximately by threshold rules when proposal costs are similar. Calibration and validation require feedback loops, ex post benchmarking (e.g., publications/citations), and sensitivity analysis in the parameters of $p(x)$ and $v(x)$.

Gross & Bergstrom (Gross et al., 2021) advance this by modeling the epistemic value of proposals under ex ante peer review. Here, reviewers compute the expected value of belief updating: $x_i \in [0,1]$0 where the divergence $x_i \in [0,1]$1 is grounded in strictly proper scoring rules, and the model quantifies risk aversion against “high-upside, high-uncertainty” proposals. Thresholds for risk acceptance emerge as

$x_i \in [0,1]$2

with high-risk projects rarely preferred unless signal-to-noise is sufficiently favorable or modified by explicit “surprise bonuses” or public-value weighting.

2. Distributed Peer Review: Workload and Accuracy Scaling

As proposal volumes rise, distributed peer review (DPR) schemes have been proposed to reduce individual reviewer burden and sharpen selection outcomes (Steppi et al., 2018). The canonical DPR pipeline is as follows:

• Each principal investigator (PI) reviews $x_i \in [0,1]$3 other proposals among a total pool $x_i \in [0,1]$4, excluding their own (balanced assignment preferred over random).
• Rankings on local review ballots are aggregated to estimate global ordering, using either Modified Borda Count (MBC): $x_i \in [0,1]$5 or Concordance Index-based Global Ranking (CIGR), which seeks

$x_i \in [0,1]$6

with $x_i \in [0,1]$7 the fraction of pairwise reviewer orderings recovered by the candidate global ranking.

Simulation studies demonstrate that CIGR outperforms MBC when reviewer error $x_i \in [0,1]$8, and that balanced pairwise coverage further improves the accuracy of both methods by up to 10 CI points. Workload–accuracy trade-offs plateau at $x_i \in [0,1]$9 reviews per PI, with multi-stage schemes (e.g., 50% eliminated after stage 1, 20% after stage 2) reducing reviewer burden by 30–50% while capturing a larger fraction of top-tier proposals.

3. AI-Driven and Agent-Based Triaging Frameworks

Recent frameworks leverage multi-agent LLM systems for automated proposal triage, particularly in large-scale settings such as telescope time allocation (Wang et al., 31 Dec 2025). AstroReview exemplifies a three-stage, modular agent pipeline:

1. Novelty and Merit: Extraction, semantic querying, and literature retrieval by specialized agents; novelty is formalized as

$v(x)$0

1. Feasibility and Yield: Instrument parameter parsing, simulator coupling, and S/N computation to assess whether requested observations are physically achievable.
2. Meta-Review and Reliability: Multiple independent LLM review agents produce rubric-based scores aggregated by a meta-reviewer; a separate verifier audits evidence, addressing hallucinations.

Empirically, introduction of automated review, reasoning trace buffers, and iterative authoring feedback loops yields sharp gains: acceptance rates of revised proposals increase by 66 percentage points after two rounds. Binary classification accuracy for final-stage accept/reject decisions reaches 87% on real-world, accepted proposal corpora.

4. Early Triage in Open-Source and Community Proposal Processes

Beyond grant funding, proposal-stage triage operates in community-driven venues such as open-source software development (Kondo et al., 8 Oct 2025). The Go language proposal workflow illustrates automated, taxonomy-driven early triage:

• Declined proposals outnumber accepted ones (41% vs. 31%), with dominant decline reasons including poor feasibility (22.8%), deprecated/duplicated suggestions, limited use cases, existing alternatives, and violation of core principles.
• A nine-category taxonomy, with inter-coder Cohen’s κ = 0.876, underpins a GPT-3.5-turbo based early classifier achieving F1 = 0.71 for decline prediction on partial comment threads.
• Practical integration involves real-time triage bots that apply the model at submission and as discussion evolves, flagging “likely-decline” proposals for rapid closure or author revision, and steering review resources toward high-value candidates.

This approach improves reviewer efficiency, shortens proposal lifecycles, and supports contributor self-screening against known failure modes.

5. Limitations of LLMs as Autonomous Triage Agents

Evaluation of current LLMs using SoundnessBench (Ho et al., 28 May 2026), a benchmark of 1,099 labeled ICLR proposals, reveals fundamental limits to AI-first triage:

• Frontier LLMs (GPT-4o, Gemini, Claude, LLaMA-3, Qwen, etc.) exhibit optimism bias in standard (justification-first) prompting: low-soundness recall averages 26% (mean false-positive rate 74%), high-soundness recall 91.8% (mean false-negative rate 8.2%), macro F1 = 54.9%.
• Switching to aggressive, conservative prompts increases rejection of low-quality proposals (low recall jumps to 80.1%) but at the cost of discarding many genuinely high-quality submissions (high recall collapses to 36.1%), with macro F1 dropping to 49.3%.
• Scale and instruction-tuning do not mitigate the tradeoff; larger models may even exacerbate optimism bias under standard prompts.
• Control experiments (adversarial flaw injection, identifier masking, audit for label leakage) confirm that LLMs can reliably detect only egregious methodological errors, missing subtler flaws. Decision thresholds are unstable and brittle, rather than reflecting internalized methodological rigor.

Present LLMs cannot serve as stand-alone “first-gate” scientific triage agents; robust deployment would require dedicated training, calibration, and retention of human-in-the-loop safeguards.

6. Practical Recommendations and Future Directions

• Threshold-based and simulation-driven triage: For standard calls, implement a calibrated workflow that (1) aggregates reviewer axis scores, (2) estimates proposal merit distributions, and (3) applies explicit innovation–score or risk–reward thresholds as in (Navarra, 2015, Steppi et al., 2018).
• DPR for large-scale or distributed panels: Employ balanced assignment, multi-stage elimination (e.g., 7 reviews per proposal in stage 1, 50% triage, then 5 in stage 2), and CIGR aggregation when review quality permits as in (Steppi et al., 2018).
• Taxonomy-enhanced triage for community proposals: Integrate proposal checklists mapping common failure reasons (duplication, feasibility, etc.) to triage logic, flagging for fast closure or candidate improvement (Kondo et al., 8 Oct 2025).
• AI/LLM-based review: Deploy LLM agents as assistants, not arbiters, ideally specializing them to structured evaluation axes and requiring explicit reasoning traces, with reliability verifiers to flag hallucinations (Wang et al., 31 Dec 2025). Do not delegate sole gatekeeping to out-of-the-box LLMs (Ho et al., 28 May 2026).
• Risk calibration: To avoid systematic neglect of high-risk/high-reward proposals, adjust triage rules to upweight information gain, implement separate risk-focused tracks, or combine private- and public-value-based reviewer scoring (Gross et al., 2021).

Taken together, a rigorous, technically grounded approach to proposal-stage scientific triage can substantially improve both innovation yield per funded proposal and efficiency of the scientific review process, provided its theoretical and empirical limitations are understood and mitigated.