- The paper introduces scBench-Long, a benchmark that assesses AI agents on recovering data-supported claims from single-cell biology using controlled answer surfaces.
- The methodology integrates diverse data modalities, structured grading, and curated distractor conditions to challenge models in multi-step scientific reasoning.
- Results reveal low pass rates and critical failure modes, emphasizing the need for continuous benchmark refinement and human expert oversight.
Verifiable Long-Horizon Benchmarking for Single-Cell Biology: scBench-Long
Introduction
scBench-Long constitutes a rigorous benchmark assessing frontier AI agents on their ability to recover complex, data-supported scientific claims from raw or minimally processed single-cell biology data. Whereas previous benchmarks in computational biology and single-cell analysis have tended to focus on local tasks or pre-digested workflows, this work foregrounds claim-conditional reasoning and authentic multi-step scientific analysis. The benchmark integrates heterogeneous evidence types, curated distractors, and deterministic endpoint grading to provide a challenging, verifiable landscape for evaluating agents across practical, research-relevant scenarios.
Benchmark Design and Construction
The scBench-Long suite spans 21 evaluations across five core biological study systems: melanoma CD8 T-cell reactivity (incorporating TCR mapping and marker-based population identification), RNA+ATAC regulatory inference for CD8 tissue-resident programs, human-monkey chimeric embryo development (including cross-species transcriptomics and ligand-receptor inference), KRAS-driven lung tumor aging, and lethal COVID-19 lung pathology. Each task is meticulously constructed around primary data modalities including single-cell or single-nucleus RNA-seq, scATAC-seq, paired immune-repertoire repertoires, ortholog mapping, developmental references, and functional validation.
A distinctive methodological advance is the adoption of structured controlled answer surfaces that mirror the core scientific objects manipulated by expert analysts: cell-state identities, donor-level compositions, regulator-direction assertions, clonotype/repertoire claims, and mechanism-level interpretations. Distractor generation and hard-fail conditions are systematically curated to penalize biologically unsupported shortcuts—e.g., canonical marker recall, abundance-based prioritization, or single-modality inference.
Candidate claims for each evaluation are only retained after successful independent reproduction from the staged data, randomized peer review, and robustness vetting across multiple model families. This protocol ensures that target conclusions are data-supported and not simply restatements of literature priors.
Evaluation Methods and Metrics
Each evaluation requires the agent to produce a structured JSON answer, strictly graded via deterministic, endpoint-based predicates (exact matching, required fields, multi-accepted synonyms, and exclusion of biologically incompatible answers). Success is measured on the final claim, with no imposed constraints on intermediate analysis paths or substep performance.
Recognizing that endpoint pass/fail grading is sparse, scBench-Long integrates task-tailored trajectory rubrics targeting single-cell analysis chokepoints—e.g., appropriate donor structure preservation, immune-repertoire integration, cross-modal regulatory reasoning, and avoidance of unjustified causal inference. These rubrics, scored independently by multiple LLM judges, provide dense (albeit imperfect) diagnostics on partial progress and failure modes, complementing but not replacing deterministic grading.
Benchmark Results
Analysis encompassed 1,068 completed trajectories across 21 evaluations and 17 model-harness pairs. The highest completed-run pass rate was achieved by Claude Opus 4.8 with Claude Code (16/63 runs, 25.4%, Wilson 95% CI: 16.3-37.3%). Gemini 3.5 Flash with the PI harness followed at 14/63 (22.2%). All other tested model-harness pairs exhibited lower performance, with run-level pass rates rarely exceeding single-digit percentages.
Task-level robustness remains limited. The best-performing system solved at least one replicate in only 8/21 evaluations and achieved all-replicate success for a given evaluation just twice. Several tasks saw zero successful completions despite numerous attempts, especially those demanding rigorous cross-modal evidence integration or causal caution.
Importantly, harness choice (e.g., PI, Claude Code, OpenAI Codex) was a significant modulator of performance. For instance, Claude Opus 4.8's success rate was higher with Claude Code versus PI; for GPT-5.4, PI outperformed Codex on endpoint pass rate, with task-level sensitivity varying across evaluations. These findings reinforce that reliable agent evaluation must consider both the foundation model and harness configuration as an inseparable system.
Diagnostic Insights: Trajectory Rubrics and Failure Modes
Though endpoint pass rates are low, rubric-based diagnostic assessment yields richer quantitative insights. Rubric judges were highly consistent (mean pairwise correlation 0.90; mean SD per trajectory 5.5 points; AUC for rubric-to-pass binary classification 0.77). High rubric scores are enriched for but not wholly predictive of endpoint passes.
The most prevalent failure modes are:
- Substitution of Prior Literature for Empirical Signal: Agents frequently anchor on familiar scientific narratives or canonical marker/phenotype relationships, even when presented with data specifically constructed to require new combinatorial reasoning or debiasing.
- Conflation of Raw Abundance with Scientific Relevance: Agents often prioritize signals or classes with the largest numerical representation (e.g., dominant ligand-receptor pairs or highly expressed genes), disregarding the need for contextual or directionality assessment.
- Erroneous Mechanistic Inference from Association: Tasks requiring causal reasoning from multifactorial data often elicit unwarranted mechanism claims from agents, highlighting poor caution in distinguishing correlation from causation.
- Failure to Integrate Modalities: Many tasks necessitating synthesis across RNA-seq and ATAC-seq, or requiring immune-repertoire and expression data to be jointly interpreted, expose a bottleneck where agents fail to combine disparate evidence layers, leading to incomplete or inaccurate answers.
Theoretical and Practical Implications
The low but nonzero success rates across all tested model-harness pairs demonstrate that compositional chain-of-thought reasoning and empirical claim recovery in high-dimensional biological data remains a frontier challenge for AI systems. Far from exhibiting broad generalization or scientific creativity, tested agents predominantly succeed via "local" or partial analysis steps, failing to achieve robust, verifiable multi-step reasoning of the kind required in authentic biomedical research.
Practically, these results caution against deploying autonomous AI agents for high-impact biological inference without human expert review. The design of scBench-Long highlights evaluation-as-specification: new agent behaviors and failure modes will necessitate continuous evolution of benchmarks, rubrics, and grading strategies. More broadly, benchmarks like scBench-Long define iterative outer loops for AI agent progress in computational biology, focusing the field on datasets, failure probes, and evidence structures encountered in real-world practice.
Future Directions
As models grow in context capacity, chain-of-thought sophistication, and multi-modal input processing, future versions of scBench-Long may need to incorporate even more diverse evidence types (e.g., spatial transcriptomics, proteomics, large-scale perturbation screens), and deepen integration with standardized scientific knowledge representations. Evaluations on dataset slices not previously published, or involving challenge sets constructed with adversarial distractors, will be critical for verifying true causal and compositional reasoning.
The dynamic interplay between model improvements and harness/system engineering observed in scBench-Long also motivates the development of more standardized agent execution environments and unified logging/interpretability protocols, as robustness and reproducibility of scientific claims hinges on system-level, not model-only, reliability.
Conclusion
scBench-Long provides a formal, verifiable challenge for AI agents operating in high-dimensional, claim-conditional single-cell biology analysis. It stands as the most comprehensive diagnostic to date for evaluating whether agents can autonomously synthesize empirical evidence into robust scientific claims. Current agents, even at the frontier, solve only a minority of such challenges, most often faltering at integrating heterogeneous evidence or resisting overfit to familiar but unsupported biological priors. Further advancement in agent systems for scientific reasoning will require transparent, verifiable, and evolving benchmarks such as scBench-Long as a disciplinary standard for progress measurement and diagnostic refinement.
